wehub-resource-sync
1682 lines
77 KiB
Plaintext
1682 lines
77 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "e99226be",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.006348,
|
||
"end_time": "2026-06-13T02:32:48.396350+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:48.390002+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"# Futures Session Aggregation: Hourly to Daily\n",
|
||
"\n",
|
||
"**Docker image**: `ml4t`\n",
|
||
"\n",
|
||
"**Purpose**: Convert hourly continuous futures bars (Databento, UTC) to\n",
|
||
"session-aware daily bars that respect the 4:00 PM Central Time CME session\n",
|
||
"boundary, applying ratio back-adjustment to eliminate roll-induced price\n",
|
||
"gaps.\n",
|
||
"\n",
|
||
"**Learning objectives**:\n",
|
||
"\n",
|
||
"- Understand why CME session dates differ from UTC calendar dates and how\n",
|
||
" bars on Sunday evening belong to Monday's session.\n",
|
||
"- Apply ratio (multiplicative) back-adjustment to a continuous series so\n",
|
||
" percentage returns are preserved across rolls.\n",
|
||
"- Aggregate hourly bars to session-correct daily OHLCV across all 30 products\n",
|
||
" and three tenors (front month, first deferred, second deferred).\n",
|
||
"\n",
|
||
"**Book reference**: §2.2 (\"The Asset-Class Market Data Landscape\" — Futures);\n",
|
||
"adjustment methodology compared in `06_futures_continuous`.\n",
|
||
"\n",
|
||
"**Prerequisites**: `data` package on `PYTHONPATH`; hourly continuous parquet\n",
|
||
"present at `ML4T_DATA_PATH/futures/market/continuous/`. See\n",
|
||
"[`06_futures_continuous`](06_futures_continuous.ipynb) for the teaching\n",
|
||
"explanation of ratio vs Panama adjustment."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 1,
|
||
"id": "d7e86a83",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:32:48.405053Z",
|
||
"iopub.status.busy": "2026-06-13T02:32:48.404810Z",
|
||
"iopub.status.idle": "2026-06-13T02:32:49.709324Z",
|
||
"shell.execute_reply": "2026-06-13T02:32:49.708787Z"
|
||
},
|
||
"lines_to_next_cell": 2,
|
||
"papermill": {
|
||
"duration": 1.311442,
|
||
"end_time": "2026-06-13T02:32:49.709794+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:48.398352+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"\"\"\"Session-aware aggregation of hourly futures to daily bars.\"\"\"\n",
|
||
"\n",
|
||
"import os\n",
|
||
"from datetime import datetime, timedelta\n",
|
||
"from zoneinfo import ZoneInfo\n",
|
||
"\n",
|
||
"import numpy as np\n",
|
||
"import polars as pl\n",
|
||
"\n",
|
||
"from data import load_cme_futures\n",
|
||
"from utils import ML4T_DATA_PATH\n",
|
||
"from utils.paths import get_chapter_dir\n",
|
||
"\n",
|
||
"# Output path for session-aggregated daily data. Default writes to a\n",
|
||
"# chapter-local directory; set WRITE_TO_DATA=1 to materialize the canonical\n",
|
||
"# daily parquet under ML4T_DATA_PATH for downstream notebooks.\n",
|
||
"WRITE_TO_DATA = os.environ.get(\"WRITE_TO_DATA\", \"0\") == \"1\"\n",
|
||
"OUTPUT_DIR = (\n",
|
||
" ML4T_DATA_PATH / \"futures\" / \"market\" / \"continuous\" / \"daily\"\n",
|
||
" if WRITE_TO_DATA\n",
|
||
" else get_chapter_dir(2) / \"output\" / \"futures_daily\"\n",
|
||
")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 2,
|
||
"id": "8975714c",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:32:49.713340Z",
|
||
"iopub.status.busy": "2026-06-13T02:32:49.713244Z",
|
||
"iopub.status.idle": "2026-06-13T02:32:49.714821Z",
|
||
"shell.execute_reply": "2026-06-13T02:32:49.714590Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.003871,
|
||
"end_time": "2026-06-13T02:32:49.715221+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:49.711350+00:00",
|
||
"status": "completed"
|
||
},
|
||
"tags": [
|
||
"parameters"
|
||
]
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Production defaults — Papermill injects overrides for CI"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "3658bd94",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.001264,
|
||
"end_time": "2026-06-13T02:32:49.717833+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:49.716569+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"## 1. CME Session Boundaries\n",
|
||
"\n",
|
||
"### Session Definition\n",
|
||
"\n",
|
||
"CME Globex sessions follow this schedule:\n",
|
||
"- **Session Start**: Sunday 5:00 PM CT (for Monday session)\n",
|
||
"- **Session End**: 4:00 PM CT (defines the session date)\n",
|
||
"- **Daily Maintenance**: 4:00-5:00 PM CT (1-hour break)\n",
|
||
"\n",
|
||
"### Why This Matters\n",
|
||
"\n",
|
||
"If we aggregate by calendar day (midnight UTC), we split a single trading\n",
|
||
"session across two days, creating incorrect daily bars:\n",
|
||
"\n",
|
||
"| Approach | Sunday 11 PM UTC | Monday 3 PM UTC |\n",
|
||
"|----------|------------------|-----------------|\n",
|
||
"| **Calendar Day (Wrong)** | Sunday | Monday |\n",
|
||
"| **CME Session (Correct)** | Monday | Monday |\n",
|
||
"\n",
|
||
"Both bars belong to Monday's session (which ends Monday 4 PM CT)."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 3,
|
||
"id": "3e24aed3",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:32:49.720934Z",
|
||
"iopub.status.busy": "2026-06-13T02:32:49.720817Z",
|
||
"iopub.status.idle": "2026-06-13T02:32:49.723277Z",
|
||
"shell.execute_reply": "2026-06-13T02:32:49.722996Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.004609,
|
||
"end_time": "2026-06-13T02:32:49.723722+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:49.719113+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Timezone constants\n",
|
||
"CT = ZoneInfo(\"America/Chicago\")\n",
|
||
"UTC = ZoneInfo(\"UTC\")\n",
|
||
"\n",
|
||
"# CME session ends at 4 PM CT\n",
|
||
"SESSION_END_HOUR_CT = 16 # 4:00 PM\n",
|
||
"\n",
|
||
"\n",
|
||
"def assign_cme_session_date(ts: datetime) -> datetime:\n",
|
||
" \"\"\"\n",
|
||
" Assign CME session date to a UTC timestamp.\n",
|
||
"\n",
|
||
" The session date is the date when the session ENDS (4 PM CT).\n",
|
||
" A bar at Sunday 11 PM UTC belongs to Monday's session.\n",
|
||
"\n",
|
||
" CME closes Friday at 4 PM CT and reopens Sunday 5 PM CT.\n",
|
||
" Bars after Friday 4 PM CT still belong to Friday's session —\n",
|
||
" they must NOT roll to Saturday.\n",
|
||
"\n",
|
||
" Args:\n",
|
||
" ts: UTC timestamp\n",
|
||
"\n",
|
||
" Returns:\n",
|
||
" Session date (as date, no time component)\n",
|
||
" \"\"\"\n",
|
||
" # Convert to Central Time\n",
|
||
" ts_ct = ts.astimezone(CT)\n",
|
||
"\n",
|
||
" # If we're past 4 PM CT, this belongs to tomorrow's session\n",
|
||
" if ts_ct.hour >= SESSION_END_HOUR_CT:\n",
|
||
" candidate = ts_ct.date() + timedelta(days=1)\n",
|
||
" # Friday after 4 PM CT → keep as Friday (no Saturday session)\n",
|
||
" # isoweekday: Mon=1, Fri=5, Sat=6\n",
|
||
" if candidate.isoweekday() == 6: # Saturday\n",
|
||
" candidate = ts_ct.date() # Keep as Friday\n",
|
||
" session_date = candidate\n",
|
||
" else:\n",
|
||
" session_date = ts_ct.date()\n",
|
||
"\n",
|
||
" return session_date"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 4,
|
||
"id": "201d3175",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:32:49.726976Z",
|
||
"iopub.status.busy": "2026-06-13T02:32:49.726911Z",
|
||
"iopub.status.idle": "2026-06-13T02:32:49.729100Z",
|
||
"shell.execute_reply": "2026-06-13T02:32:49.728816Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.004225,
|
||
"end_time": "2026-06-13T02:32:49.729466+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:49.725241+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Session Assignment Examples:\n",
|
||
" 2024-01-07 23:00:00+00:00 (Sun 05:00 PM CT) -> Session: 2024-01-08\n",
|
||
" 2024-01-08 15:00:00+00:00 (Mon 09:00 AM CT) -> Session: 2024-01-08\n",
|
||
" 2024-01-08 22:00:00+00:00 (Mon 04:00 PM CT) -> Session: 2024-01-09\n",
|
||
" 2024-01-12 22:00:00+00:00 (Fri 04:00 PM CT) -> Session: 2024-01-12\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Quick test\n",
|
||
"test_times = [\n",
|
||
" datetime(2024, 1, 7, 23, 0, tzinfo=UTC), # Sunday 11 PM UTC = Sunday 5 PM CT -> Monday\n",
|
||
" datetime(2024, 1, 8, 15, 0, tzinfo=UTC), # Monday 3 PM UTC = Monday 9 AM CT -> Monday\n",
|
||
" datetime(2024, 1, 8, 22, 0, tzinfo=UTC), # Monday 10 PM UTC = Monday 4 PM CT -> Tuesday\n",
|
||
" datetime(\n",
|
||
" 2024, 1, 12, 22, 0, tzinfo=UTC\n",
|
||
" ), # Friday 10 PM UTC = Friday 4 PM CT -> Friday (NOT Saturday)\n",
|
||
"]\n",
|
||
"\n",
|
||
"print(\"Session Assignment Examples:\")\n",
|
||
"for ts in test_times:\n",
|
||
" ts_ct = ts.astimezone(CT)\n",
|
||
" session = assign_cme_session_date(ts)\n",
|
||
" print(f\" {ts} ({ts_ct.strftime('%a %I:%M %p CT')}) -> Session: {session}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "b3e5f800",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.001307,
|
||
"end_time": "2026-06-13T02:32:49.732168+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:49.730861+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"## 2. Load Hourly Continuous Data\n",
|
||
"\n",
|
||
"We load all products and tenors from the DataBento hourly data."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 5,
|
||
"id": "a4ff9fe9",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:32:49.735366Z",
|
||
"iopub.status.busy": "2026-06-13T02:32:49.735291Z",
|
||
"iopub.status.idle": "2026-06-13T02:32:49.938971Z",
|
||
"shell.execute_reply": "2026-06-13T02:32:49.938527Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.206196,
|
||
"end_time": "2026-06-13T02:32:49.939687+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:49.733491+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Loaded 5,463,741 hourly bars\n",
|
||
"Products: 30\n"
|
||
]
|
||
},
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Tenors: [0, 1, 2]\n",
|
||
"Date range: 2011-01-02 23:00:00+00:00 to 2025-12-30 23:00:00+00:00\n",
|
||
"Available products: 6A, 6B, 6C, 6E, 6J, 6S, CL, ES, GC, GF, HE, HG, HO, LE, NG, NQ, PL, RB, RTY, SI, YM, ZB, ZC, ZF, ZL, ZM, ZN, ZS, ZT, ZW\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"hourly = load_cme_futures(continuous=True, frequency=\"hourly\")\n",
|
||
"products = sorted(hourly[\"product\"].unique().to_list())\n",
|
||
"\n",
|
||
"print(f\"Loaded {len(hourly):,} hourly bars\")\n",
|
||
"print(f\"Products: {hourly['product'].n_unique()}\")\n",
|
||
"print(f\"Tenors: {sorted(hourly['tenor'].unique().to_list())}\")\n",
|
||
"print(f\"Date range: {hourly['timestamp'].min()} to {hourly['timestamp'].max()}\")\n",
|
||
"print(f\"Available products: {', '.join(products)}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 6,
|
||
"id": "b053036c",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:32:49.943703Z",
|
||
"iopub.status.busy": "2026-06-13T02:32:49.943599Z",
|
||
"iopub.status.idle": "2026-06-13T02:32:49.956356Z",
|
||
"shell.execute_reply": "2026-06-13T02:32:49.955944Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.015263,
|
||
"end_time": "2026-06-13T02:32:49.956663+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:49.941400+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/html": [
|
||
"<div><style>\n",
|
||
".dataframe > thead > tr,\n",
|
||
".dataframe > tbody > tr {\n",
|
||
" text-align: right;\n",
|
||
" white-space: pre-wrap;\n",
|
||
"}\n",
|
||
"</style>\n",
|
||
"<small>shape: (10, 8)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>timestamp</th><th>product</th><th>tenor</th><th>open</th><th>high</th><th>low</th><th>close</th><th>volume</th></tr><tr><td>datetime[ns, UTC]</td><td>str</td><td>i8</td><td>f64</td><td>f64</td><td>f64</td><td>f64</td><td>u64</td></tr></thead><tbody><tr><td>2011-01-02 23:00:00 UTC</td><td>"ES"</td><td>0</td><td>1256.0</td><td>1257.75</td><td>1255.25</td><td>1256.75</td><td>7479</td></tr><tr><td>2011-01-02 23:00:00 UTC</td><td>"ES"</td><td>1</td><td>1250.75</td><td>1250.75</td><td>1250.75</td><td>1250.75</td><td>5</td></tr><tr><td>2011-01-02 23:00:00 UTC</td><td>"ES"</td><td>2</td><td>1248.0</td><td>1248.0</td><td>1248.0</td><td>1248.0</td><td>5</td></tr><tr><td>2011-01-03 00:00:00 UTC</td><td>"ES"</td><td>0</td><td>1256.75</td><td>1257.5</td><td>1255.75</td><td>1257.0</td><td>2715</td></tr><tr><td>2011-01-03 00:00:00 UTC</td><td>"ES"</td><td>1</td><td>1251.5</td><td>1251.5</td><td>1251.5</td><td>1251.5</td><td>3</td></tr><tr><td>2011-01-03 01:00:00 UTC</td><td>"ES"</td><td>0</td><td>1257.0</td><td>1257.25</td><td>1256.75</td><td>1257.0</td><td>2095</td></tr><tr><td>2011-01-03 01:00:00 UTC</td><td>"ES"</td><td>1</td><td>1252.0</td><td>1252.25</td><td>1252.0</td><td>1252.25</td><td>2</td></tr><tr><td>2011-01-03 02:00:00 UTC</td><td>"ES"</td><td>0</td><td>1257.0</td><td>1257.5</td><td>1256.75</td><td>1257.25</td><td>1556</td></tr><tr><td>2011-01-03 03:00:00 UTC</td><td>"ES"</td><td>0</td><td>1257.25</td><td>1261.0</td><td>1257.25</td><td>1260.25</td><td>6798</td></tr><tr><td>2011-01-03 03:00:00 UTC</td><td>"ES"</td><td>1</td><td>1252.25</td><td>1255.5</td><td>1252.25</td><td>1254.75</td><td>53</td></tr></tbody></table></div>"
|
||
],
|
||
"text/plain": [
|
||
"shape: (10, 8)\n",
|
||
"┌─────────────────────────┬─────────┬───────┬─────────┬─────────┬─────────┬─────────┬────────┐\n",
|
||
"│ timestamp ┆ product ┆ tenor ┆ open ┆ high ┆ low ┆ close ┆ volume │\n",
|
||
"│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │\n",
|
||
"│ datetime[ns, UTC] ┆ str ┆ i8 ┆ f64 ┆ f64 ┆ f64 ┆ f64 ┆ u64 │\n",
|
||
"╞═════════════════════════╪═════════╪═══════╪═════════╪═════════╪═════════╪═════════╪════════╡\n",
|
||
"│ 2011-01-02 23:00:00 UTC ┆ ES ┆ 0 ┆ 1256.0 ┆ 1257.75 ┆ 1255.25 ┆ 1256.75 ┆ 7479 │\n",
|
||
"│ 2011-01-02 23:00:00 UTC ┆ ES ┆ 1 ┆ 1250.75 ┆ 1250.75 ┆ 1250.75 ┆ 1250.75 ┆ 5 │\n",
|
||
"│ 2011-01-02 23:00:00 UTC ┆ ES ┆ 2 ┆ 1248.0 ┆ 1248.0 ┆ 1248.0 ┆ 1248.0 ┆ 5 │\n",
|
||
"│ 2011-01-03 00:00:00 UTC ┆ ES ┆ 0 ┆ 1256.75 ┆ 1257.5 ┆ 1255.75 ┆ 1257.0 ┆ 2715 │\n",
|
||
"│ 2011-01-03 00:00:00 UTC ┆ ES ┆ 1 ┆ 1251.5 ┆ 1251.5 ┆ 1251.5 ┆ 1251.5 ┆ 3 │\n",
|
||
"│ 2011-01-03 01:00:00 UTC ┆ ES ┆ 0 ┆ 1257.0 ┆ 1257.25 ┆ 1256.75 ┆ 1257.0 ┆ 2095 │\n",
|
||
"│ 2011-01-03 01:00:00 UTC ┆ ES ┆ 1 ┆ 1252.0 ┆ 1252.25 ┆ 1252.0 ┆ 1252.25 ┆ 2 │\n",
|
||
"│ 2011-01-03 02:00:00 UTC ┆ ES ┆ 0 ┆ 1257.0 ┆ 1257.5 ┆ 1256.75 ┆ 1257.25 ┆ 1556 │\n",
|
||
"│ 2011-01-03 03:00:00 UTC ┆ ES ┆ 0 ┆ 1257.25 ┆ 1261.0 ┆ 1257.25 ┆ 1260.25 ┆ 6798 │\n",
|
||
"│ 2011-01-03 03:00:00 UTC ┆ ES ┆ 1 ┆ 1252.25 ┆ 1255.5 ┆ 1252.25 ┆ 1254.75 ┆ 53 │\n",
|
||
"└─────────────────────────┴─────────┴───────┴─────────┴─────────┴─────────┴─────────┴────────┘"
|
||
]
|
||
},
|
||
"execution_count": 6,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"hourly.filter(pl.col(\"product\") == \"ES\").select(\n",
|
||
" \"timestamp\", \"product\", \"tenor\", \"open\", \"high\", \"low\", \"close\", \"volume\"\n",
|
||
").head(10)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "789ab111",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.00198,
|
||
"end_time": "2026-06-13T02:32:49.960933+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:49.958953+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"## 3. Assign Session Dates\n",
|
||
"\n",
|
||
"We add a `session_date` column using Polars expressions for efficiency."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 7,
|
||
"id": "c2d2cf29",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:32:49.964508Z",
|
||
"iopub.status.busy": "2026-06-13T02:32:49.964418Z",
|
||
"iopub.status.idle": "2026-06-13T02:32:49.966811Z",
|
||
"shell.execute_reply": "2026-06-13T02:32:49.966525Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.004606,
|
||
"end_time": "2026-06-13T02:32:49.967077+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:49.962471+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Vectorized session date assignment using Polars\n",
|
||
"# Convert to Central Time, then check if hour >= 16 (4 PM)\n",
|
||
"\n",
|
||
"\n",
|
||
"def add_session_date(df: pl.DataFrame) -> pl.DataFrame:\n",
|
||
" \"\"\"Add session_date column based on CME session boundaries.\n",
|
||
"\n",
|
||
" Friday after 4 PM CT stays as Friday — CME has no Saturday session.\n",
|
||
" \"\"\"\n",
|
||
" return (\n",
|
||
" df.with_columns(pl.col(\"timestamp\").dt.convert_time_zone(\"America/Chicago\").alias(\"ts_ct\"))\n",
|
||
" .with_columns(\n",
|
||
" pl.col(\"ts_ct\").dt.date().alias(\"_ct_date\"),\n",
|
||
" (pl.col(\"ts_ct\").dt.hour() >= SESSION_END_HOUR_CT).alias(\"_after_close\"),\n",
|
||
" # isoweekday: Mon=1 ... Fri=5, Sat=6, Sun=7\n",
|
||
" (pl.col(\"ts_ct\").dt.weekday() == 5).alias(\"_is_friday\"),\n",
|
||
" )\n",
|
||
" .with_columns(\n",
|
||
" # After 4 PM CT → next day, UNLESS it's Friday (no Saturday session)\n",
|
||
" pl.when(pl.col(\"_after_close\") & ~pl.col(\"_is_friday\"))\n",
|
||
" .then(pl.col(\"_ct_date\") + pl.duration(days=1))\n",
|
||
" .otherwise(pl.col(\"_ct_date\"))\n",
|
||
" .alias(\"session_date\")\n",
|
||
" )\n",
|
||
" .drop(\"ts_ct\", \"_ct_date\", \"_after_close\", \"_is_friday\")\n",
|
||
" )"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 8,
|
||
"id": "270ed44a",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:32:49.970401Z",
|
||
"iopub.status.busy": "2026-06-13T02:32:49.970340Z",
|
||
"iopub.status.idle": "2026-06-13T02:32:50.056488Z",
|
||
"shell.execute_reply": "2026-06-13T02:32:50.056160Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.088448,
|
||
"end_time": "2026-06-13T02:32:50.056962+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:49.968514+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Session dates assigned (ES sample):\n"
|
||
]
|
||
},
|
||
{
|
||
"data": {
|
||
"text/html": [
|
||
"<div><style>\n",
|
||
".dataframe > thead > tr,\n",
|
||
".dataframe > tbody > tr {\n",
|
||
" text-align: right;\n",
|
||
" white-space: pre-wrap;\n",
|
||
"}\n",
|
||
"</style>\n",
|
||
"<small>shape: (15, 6)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>timestamp</th><th>session_date</th><th>product</th><th>tenor</th><th>close</th><th>volume</th></tr><tr><td>datetime[ns, UTC]</td><td>date</td><td>str</td><td>i8</td><td>f64</td><td>u64</td></tr></thead><tbody><tr><td>2011-01-02 23:00:00 UTC</td><td>2011-01-03</td><td>"ES"</td><td>0</td><td>1256.75</td><td>7479</td></tr><tr><td>2011-01-02 23:00:00 UTC</td><td>2011-01-03</td><td>"ES"</td><td>1</td><td>1250.75</td><td>5</td></tr><tr><td>2011-01-02 23:00:00 UTC</td><td>2011-01-03</td><td>"ES"</td><td>2</td><td>1248.0</td><td>5</td></tr><tr><td>2011-01-03 00:00:00 UTC</td><td>2011-01-03</td><td>"ES"</td><td>0</td><td>1257.0</td><td>2715</td></tr><tr><td>2011-01-03 00:00:00 UTC</td><td>2011-01-03</td><td>"ES"</td><td>1</td><td>1251.5</td><td>3</td></tr><tr><td>…</td><td>…</td><td>…</td><td>…</td><td>…</td><td>…</td></tr><tr><td>2011-01-03 03:00:00 UTC</td><td>2011-01-03</td><td>"ES"</td><td>2</td><td>1249.5</td><td>5</td></tr><tr><td>2011-01-03 04:00:00 UTC</td><td>2011-01-03</td><td>"ES"</td><td>0</td><td>1260.25</td><td>2696</td></tr><tr><td>2011-01-03 04:00:00 UTC</td><td>2011-01-03</td><td>"ES"</td><td>1</td><td>1255.5</td><td>27</td></tr><tr><td>2011-01-03 05:00:00 UTC</td><td>2011-01-03</td><td>"ES"</td><td>0</td><td>1260.0</td><td>937</td></tr><tr><td>2011-01-03 05:00:00 UTC</td><td>2011-01-03</td><td>"ES"</td><td>1</td><td>1255.0</td><td>1</td></tr></tbody></table></div>"
|
||
],
|
||
"text/plain": [
|
||
"shape: (15, 6)\n",
|
||
"┌─────────────────────────┬──────────────┬─────────┬───────┬─────────┬────────┐\n",
|
||
"│ timestamp ┆ session_date ┆ product ┆ tenor ┆ close ┆ volume │\n",
|
||
"│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │\n",
|
||
"│ datetime[ns, UTC] ┆ date ┆ str ┆ i8 ┆ f64 ┆ u64 │\n",
|
||
"╞═════════════════════════╪══════════════╪═════════╪═══════╪═════════╪════════╡\n",
|
||
"│ 2011-01-02 23:00:00 UTC ┆ 2011-01-03 ┆ ES ┆ 0 ┆ 1256.75 ┆ 7479 │\n",
|
||
"│ 2011-01-02 23:00:00 UTC ┆ 2011-01-03 ┆ ES ┆ 1 ┆ 1250.75 ┆ 5 │\n",
|
||
"│ 2011-01-02 23:00:00 UTC ┆ 2011-01-03 ┆ ES ┆ 2 ┆ 1248.0 ┆ 5 │\n",
|
||
"│ 2011-01-03 00:00:00 UTC ┆ 2011-01-03 ┆ ES ┆ 0 ┆ 1257.0 ┆ 2715 │\n",
|
||
"│ 2011-01-03 00:00:00 UTC ┆ 2011-01-03 ┆ ES ┆ 1 ┆ 1251.5 ┆ 3 │\n",
|
||
"│ … ┆ … ┆ … ┆ … ┆ … ┆ … │\n",
|
||
"│ 2011-01-03 03:00:00 UTC ┆ 2011-01-03 ┆ ES ┆ 2 ┆ 1249.5 ┆ 5 │\n",
|
||
"│ 2011-01-03 04:00:00 UTC ┆ 2011-01-03 ┆ ES ┆ 0 ┆ 1260.25 ┆ 2696 │\n",
|
||
"│ 2011-01-03 04:00:00 UTC ┆ 2011-01-03 ┆ ES ┆ 1 ┆ 1255.5 ┆ 27 │\n",
|
||
"│ 2011-01-03 05:00:00 UTC ┆ 2011-01-03 ┆ ES ┆ 0 ┆ 1260.0 ┆ 937 │\n",
|
||
"│ 2011-01-03 05:00:00 UTC ┆ 2011-01-03 ┆ ES ┆ 1 ┆ 1255.0 ┆ 1 │\n",
|
||
"└─────────────────────────┴──────────────┴─────────┴───────┴─────────┴────────┘"
|
||
]
|
||
},
|
||
"execution_count": 8,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"hourly_with_sessions = add_session_date(hourly)\n",
|
||
"\n",
|
||
"print(\"Session dates assigned (ES sample):\")\n",
|
||
"hourly_with_sessions.filter(pl.col(\"product\") == \"ES\").select(\n",
|
||
" \"timestamp\", \"session_date\", \"product\", \"tenor\", \"close\", \"volume\"\n",
|
||
").head(15)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "8fd833ce",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.001509,
|
||
"end_time": "2026-06-13T02:32:50.060141+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:50.058632+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"Walk a single calendar day for the ES front month: bars with `timestamp <\n",
|
||
"2024-01-08 22:00 UTC` carry session_date 2024-01-08; bars at or after 22:00\n",
|
||
"UTC (= 16:00 CT, the close) carry session_date 2024-01-09."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 9,
|
||
"id": "e5c11997",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:32:50.063608Z",
|
||
"iopub.status.busy": "2026-06-13T02:32:50.063528Z",
|
||
"iopub.status.idle": "2026-06-13T02:32:50.075621Z",
|
||
"shell.execute_reply": "2026-06-13T02:32:50.075309Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.014315,
|
||
"end_time": "2026-06-13T02:32:50.075937+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:50.061622+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/html": [
|
||
"<div><style>\n",
|
||
".dataframe > thead > tr,\n",
|
||
".dataframe > tbody > tr {\n",
|
||
" text-align: right;\n",
|
||
" white-space: pre-wrap;\n",
|
||
"}\n",
|
||
"</style>\n",
|
||
"<small>shape: (23, 4)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>timestamp</th><th>session_date</th><th>close</th><th>volume</th></tr><tr><td>datetime[ns, UTC]</td><td>date</td><td>f64</td><td>u64</td></tr></thead><tbody><tr><td>2024-01-08 00:00:00 UTC</td><td>2024-01-08</td><td>4741.25</td><td>4802</td></tr><tr><td>2024-01-08 01:00:00 UTC</td><td>2024-01-08</td><td>4736.5</td><td>7702</td></tr><tr><td>2024-01-08 02:00:00 UTC</td><td>2024-01-08</td><td>4734.75</td><td>6587</td></tr><tr><td>2024-01-08 03:00:00 UTC</td><td>2024-01-08</td><td>4731.25</td><td>6239</td></tr><tr><td>2024-01-08 04:00:00 UTC</td><td>2024-01-08</td><td>4732.25</td><td>2230</td></tr><tr><td>…</td><td>…</td><td>…</td><td>…</td></tr><tr><td>2024-01-08 18:00:00 UTC</td><td>2024-01-08</td><td>4775.5</td><td>74155</td></tr><tr><td>2024-01-08 19:00:00 UTC</td><td>2024-01-08</td><td>4793.5</td><td>151395</td></tr><tr><td>2024-01-08 20:00:00 UTC</td><td>2024-01-08</td><td>4801.5</td><td>274626</td></tr><tr><td>2024-01-08 21:00:00 UTC</td><td>2024-01-08</td><td>4798.0</td><td>98789</td></tr><tr><td>2024-01-08 23:00:00 UTC</td><td>2024-01-09</td><td>4797.25</td><td>5367</td></tr></tbody></table></div>"
|
||
],
|
||
"text/plain": [
|
||
"shape: (23, 4)\n",
|
||
"┌─────────────────────────┬──────────────┬─────────┬────────┐\n",
|
||
"│ timestamp ┆ session_date ┆ close ┆ volume │\n",
|
||
"│ --- ┆ --- ┆ --- ┆ --- │\n",
|
||
"│ datetime[ns, UTC] ┆ date ┆ f64 ┆ u64 │\n",
|
||
"╞═════════════════════════╪══════════════╪═════════╪════════╡\n",
|
||
"│ 2024-01-08 00:00:00 UTC ┆ 2024-01-08 ┆ 4741.25 ┆ 4802 │\n",
|
||
"│ 2024-01-08 01:00:00 UTC ┆ 2024-01-08 ┆ 4736.5 ┆ 7702 │\n",
|
||
"│ 2024-01-08 02:00:00 UTC ┆ 2024-01-08 ┆ 4734.75 ┆ 6587 │\n",
|
||
"│ 2024-01-08 03:00:00 UTC ┆ 2024-01-08 ┆ 4731.25 ┆ 6239 │\n",
|
||
"│ 2024-01-08 04:00:00 UTC ┆ 2024-01-08 ┆ 4732.25 ┆ 2230 │\n",
|
||
"│ … ┆ … ┆ … ┆ … │\n",
|
||
"│ 2024-01-08 18:00:00 UTC ┆ 2024-01-08 ┆ 4775.5 ┆ 74155 │\n",
|
||
"│ 2024-01-08 19:00:00 UTC ┆ 2024-01-08 ┆ 4793.5 ┆ 151395 │\n",
|
||
"│ 2024-01-08 20:00:00 UTC ┆ 2024-01-08 ┆ 4801.5 ┆ 274626 │\n",
|
||
"│ 2024-01-08 21:00:00 UTC ┆ 2024-01-08 ┆ 4798.0 ┆ 98789 │\n",
|
||
"│ 2024-01-08 23:00:00 UTC ┆ 2024-01-09 ┆ 4797.25 ┆ 5367 │\n",
|
||
"└─────────────────────────┴──────────────┴─────────┴────────┘"
|
||
]
|
||
},
|
||
"execution_count": 9,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"(\n",
|
||
" hourly_with_sessions.filter(\n",
|
||
" (pl.col(\"product\") == \"ES\")\n",
|
||
" & (pl.col(\"tenor\") == 0)\n",
|
||
" & (pl.col(\"timestamp\").dt.date() == pl.lit(\"2024-01-08\").str.to_date())\n",
|
||
" )\n",
|
||
" .sort(\"timestamp\")\n",
|
||
" .select(\"timestamp\", \"session_date\", \"close\", \"volume\")\n",
|
||
")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "e75e3187",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.00155,
|
||
"end_time": "2026-06-13T02:32:50.079194+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:50.077644+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"## 3b. Ratio Back-Adjustment\n",
|
||
"\n",
|
||
"Databento's continuous contracts are **unadjusted** — price gaps at roll transitions\n",
|
||
"produce spurious returns (e.g., ES Mar 2020: -11.08% artificial gap). We apply\n",
|
||
"**ratio (multiplicative)** back-adjustment using `instrument_id` to detect roll points:\n",
|
||
"\n",
|
||
"1. Detect where `instrument_id` changes between adjacent hourly bars\n",
|
||
"2. Compute ratio = new contract open / old contract close at each roll\n",
|
||
"3. Accumulate ratios backward (most recent prices stay unadjusted)\n",
|
||
"4. Multiply all OHLC prices by cumulative ratio\n",
|
||
"\n",
|
||
"Ratio adjustment preserves **percentage returns** (critical for IC, momentum features,\n",
|
||
"and backtesting) unlike Panama (additive) which distorts returns for old data and can\n",
|
||
"push prices negative for commodities with large cumulative adjustments.\n",
|
||
"\n",
|
||
"See [`06_futures_continuous`](06_futures_continuous.ipynb) for a teaching explanation of adjustment methods."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 10,
|
||
"id": "340c9f6b",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:32:50.082823Z",
|
||
"iopub.status.busy": "2026-06-13T02:32:50.082724Z",
|
||
"iopub.status.idle": "2026-06-13T02:32:50.596050Z",
|
||
"shell.execute_reply": "2026-06-13T02:32:50.595733Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.515599,
|
||
"end_time": "2026-06-13T02:32:50.596342+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:50.080743+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Roll transitions detected: 31993\n",
|
||
"Products with rolls: 30\n",
|
||
"ES roll ratios (427 rolls):\n"
|
||
]
|
||
},
|
||
{
|
||
"data": {
|
||
"text/html": [
|
||
"<div><style>\n",
|
||
".dataframe > thead > tr,\n",
|
||
".dataframe > tbody > tr {\n",
|
||
" text-align: right;\n",
|
||
" white-space: pre-wrap;\n",
|
||
"}\n",
|
||
"</style>\n",
|
||
"<small>shape: (10, 2)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>timestamp</th><th>ratio</th></tr><tr><td>datetime[ns, UTC]</td><td>f64</td></tr></thead><tbody><tr><td>2011-01-12 01:00:00 UTC</td><td>1.004164</td></tr><tr><td>2011-01-12 08:00:00 UTC</td><td>0.999407</td></tr><tr><td>2011-01-13 01:00:00 UTC</td><td>1.003929</td></tr><tr><td>2011-01-13 15:00:00 UTC</td><td>0.994914</td></tr><tr><td>2011-01-27 14:00:00 UTC</td><td>1.003525</td></tr><tr><td>2011-01-28 16:00:00 UTC</td><td>0.990647</td></tr><tr><td>2011-02-04 13:00:00 UTC</td><td>0.997876</td></tr><tr><td>2011-02-07 15:00:00 UTC</td><td>1.00948</td></tr><tr><td>2011-03-14 00:00:00 UTC</td><td>0.99653</td></tr><tr><td>2011-03-14 00:00:00 UTC</td><td>1.003676</td></tr></tbody></table></div>"
|
||
],
|
||
"text/plain": [
|
||
"shape: (10, 2)\n",
|
||
"┌─────────────────────────┬──────────┐\n",
|
||
"│ timestamp ┆ ratio │\n",
|
||
"│ --- ┆ --- │\n",
|
||
"│ datetime[ns, UTC] ┆ f64 │\n",
|
||
"╞═════════════════════════╪══════════╡\n",
|
||
"│ 2011-01-12 01:00:00 UTC ┆ 1.004164 │\n",
|
||
"│ 2011-01-12 08:00:00 UTC ┆ 0.999407 │\n",
|
||
"│ 2011-01-13 01:00:00 UTC ┆ 1.003929 │\n",
|
||
"│ 2011-01-13 15:00:00 UTC ┆ 0.994914 │\n",
|
||
"│ 2011-01-27 14:00:00 UTC ┆ 1.003525 │\n",
|
||
"│ 2011-01-28 16:00:00 UTC ┆ 0.990647 │\n",
|
||
"│ 2011-02-04 13:00:00 UTC ┆ 0.997876 │\n",
|
||
"│ 2011-02-07 15:00:00 UTC ┆ 1.00948 │\n",
|
||
"│ 2011-03-14 00:00:00 UTC ┆ 0.99653 │\n",
|
||
"│ 2011-03-14 00:00:00 UTC ┆ 1.003676 │\n",
|
||
"└─────────────────────────┴──────────┘"
|
||
]
|
||
},
|
||
"execution_count": 10,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"# Sort and detect roll transitions per (product, tenor)\n",
|
||
"hourly_sorted = hourly_with_sessions.sort([\"product\", \"tenor\", \"timestamp\"])\n",
|
||
"\n",
|
||
"# Detect instrument_id changes within each (product, tenor) group\n",
|
||
"hourly_sorted = hourly_sorted.with_columns(\n",
|
||
" pl.col(\"instrument_id\").shift(1).over(\"product\", \"tenor\").alias(\"_prev_instrument_id\"),\n",
|
||
" pl.col(\"close\").shift(1).over(\"product\", \"tenor\").alias(\"_prev_close\"),\n",
|
||
")\n",
|
||
"\n",
|
||
"# Roll points: where instrument_id changes (excluding first row of each group)\n",
|
||
"rolls = hourly_sorted.filter(\n",
|
||
" pl.col(\"_prev_instrument_id\").is_not_null()\n",
|
||
" & (pl.col(\"instrument_id\") != pl.col(\"_prev_instrument_id\"))\n",
|
||
")\n",
|
||
"\n",
|
||
"# Ratio = new contract's open / old contract's close (adjacent hourly bars)\n",
|
||
"roll_ratios = rolls.select(\n",
|
||
" \"product\",\n",
|
||
" \"tenor\",\n",
|
||
" \"timestamp\",\n",
|
||
" (pl.col(\"open\") / pl.col(\"_prev_close\")).alias(\"ratio\"),\n",
|
||
")\n",
|
||
"\n",
|
||
"print(f\"Roll transitions detected: {len(roll_ratios)}\")\n",
|
||
"print(f\"Products with rolls: {roll_ratios['product'].n_unique()}\")\n",
|
||
"\n",
|
||
"es_rolls = roll_ratios.filter(pl.col(\"product\") == \"ES\").sort(\"timestamp\")\n",
|
||
"print(f\"ES roll ratios ({len(es_rolls)} rolls):\")\n",
|
||
"es_rolls.select(\"timestamp\", \"ratio\").head(10)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "c2dc3fbf",
|
||
"metadata": {
|
||
"lines_to_next_cell": 2,
|
||
"papermill": {
|
||
"duration": 0.001586,
|
||
"end_time": "2026-06-13T02:32:50.599760+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:50.598174+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"### Ratio Back-Adjustment Function\n",
|
||
"\n",
|
||
"Walk backward through each (product, tenor) group, accumulating roll ratios to\n",
|
||
"build a cumulative multiplier for all OHLC prices."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 11,
|
||
"id": "c8718284",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:32:50.603421Z",
|
||
"iopub.status.busy": "2026-06-13T02:32:50.603339Z",
|
||
"iopub.status.idle": "2026-06-13T02:32:50.605793Z",
|
||
"shell.execute_reply": "2026-06-13T02:32:50.605526Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.004748,
|
||
"end_time": "2026-06-13T02:32:50.606113+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:50.601365+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"def ratio_adjust(group: pl.DataFrame) -> pl.DataFrame:\n",
|
||
" \"\"\"Apply ratio back-adjustment to a single (product, tenor) group.\"\"\"\n",
|
||
" group = group.sort(\"timestamp\")\n",
|
||
"\n",
|
||
" # Get roll ratios for this group\n",
|
||
" group_rolls = roll_ratios.filter(\n",
|
||
" (pl.col(\"product\") == group[\"product\"][0]) & (pl.col(\"tenor\") == group[\"tenor\"][0])\n",
|
||
" ).select(\"timestamp\", \"ratio\")\n",
|
||
"\n",
|
||
" if len(group_rolls) == 0:\n",
|
||
" return group.with_columns(pl.lit(1.0).alias(\"_cumulative_ratio\"))\n",
|
||
"\n",
|
||
" # Join roll ratios\n",
|
||
" group = group.join(group_rolls, on=\"timestamp\", how=\"left\").with_columns(\n",
|
||
" pl.col(\"ratio\").fill_null(1.0)\n",
|
||
" )\n",
|
||
"\n",
|
||
" # Cumulative ratio: product of all FUTURE ratios (reverse cumprod)\n",
|
||
" # Bars BEFORE a roll get multiplied; bars ON and AFTER the roll do not\n",
|
||
" n = len(group)\n",
|
||
" ratios = group[\"ratio\"].to_numpy()\n",
|
||
" adj = np.ones(n)\n",
|
||
" cumulative = 1.0\n",
|
||
" for i in range(n - 1, -1, -1):\n",
|
||
" adj[i] = cumulative\n",
|
||
" if ratios[i] != 1.0:\n",
|
||
" cumulative *= ratios[i]\n",
|
||
"\n",
|
||
" return group.with_columns(pl.Series(\"_cumulative_ratio\", adj)).drop(\"ratio\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 12,
|
||
"id": "10f45231",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:32:50.609671Z",
|
||
"iopub.status.busy": "2026-06-13T02:32:50.609606Z",
|
||
"iopub.status.idle": "2026-06-13T02:32:56.321091Z",
|
||
"shell.execute_reply": "2026-06-13T02:32:56.320704Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 5.713815,
|
||
"end_time": "2026-06-13T02:32:56.321497+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:50.607682+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
" Adjusted 30/90 groups\n"
|
||
]
|
||
},
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
" Adjusted 60/90 groups\n"
|
||
]
|
||
},
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
" Adjusted 90/90 groups\n",
|
||
"\n",
|
||
"Ratio adjustment applied to 5,463,741 hourly bars\n",
|
||
"ES front month cumulative ratio range: 0.8714 to 1.1592\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Apply per group\n",
|
||
"adjusted_groups = []\n",
|
||
"products_tenors = hourly_sorted.select(\"product\", \"tenor\").unique().sort(\"product\", \"tenor\")\n",
|
||
"n_groups = len(products_tenors)\n",
|
||
"\n",
|
||
"for i, row in enumerate(products_tenors.iter_rows(named=True)):\n",
|
||
" group = hourly_sorted.filter(\n",
|
||
" (pl.col(\"product\") == row[\"product\"]) & (pl.col(\"tenor\") == row[\"tenor\"])\n",
|
||
" )\n",
|
||
" adjusted = ratio_adjust(group)\n",
|
||
" adjusted_groups.append(adjusted)\n",
|
||
" if (i + 1) % 30 == 0 or i == n_groups - 1:\n",
|
||
" print(f\" Adjusted {i + 1}/{n_groups} groups\")\n",
|
||
"\n",
|
||
"hourly_adjusted = pl.concat(adjusted_groups)\n",
|
||
"\n",
|
||
"# Apply ratio adjustment to OHLC (multiply, not add)\n",
|
||
"hourly_adjusted = hourly_adjusted.with_columns(\n",
|
||
" (pl.col(\"open\") * pl.col(\"_cumulative_ratio\")).alias(\"open\"),\n",
|
||
" (pl.col(\"high\") * pl.col(\"_cumulative_ratio\")).alias(\"high\"),\n",
|
||
" (pl.col(\"low\") * pl.col(\"_cumulative_ratio\")).alias(\"low\"),\n",
|
||
" (pl.col(\"close\") * pl.col(\"_cumulative_ratio\")).alias(\"close\"),\n",
|
||
")\n",
|
||
"\n",
|
||
"print(f\"\\nRatio adjustment applied to {len(hourly_adjusted):,} hourly bars\")\n",
|
||
"\n",
|
||
"# Show adjustment magnitude for ES front month\n",
|
||
"es_adj = hourly_adjusted.filter((pl.col(\"product\") == \"ES\") & (pl.col(\"tenor\") == 0)).sort(\n",
|
||
" \"timestamp\"\n",
|
||
")\n",
|
||
"print(\n",
|
||
" f\"ES front month cumulative ratio range: \"\n",
|
||
" f\"{es_adj['_cumulative_ratio'].min():.4f} to {es_adj['_cumulative_ratio'].max():.4f}\"\n",
|
||
")\n",
|
||
"\n",
|
||
"# Replace hourly_with_sessions with adjusted data for downstream aggregation\n",
|
||
"hourly_with_sessions = hourly_adjusted.drop(\n",
|
||
" \"_prev_instrument_id\", \"_prev_close\", \"_cumulative_ratio\"\n",
|
||
")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "6d37ed65",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.001635,
|
||
"end_time": "2026-06-13T02:32:56.325188+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:56.323553+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"## 4. Aggregate to Daily OHLCV\n",
|
||
"\n",
|
||
"Aggregate hourly bars to daily using session boundaries:\n",
|
||
"- **Open**: First bar's open (ratio-adjusted)\n",
|
||
"- **High**: Maximum high (ratio-adjusted)\n",
|
||
"- **Low**: Minimum low (ratio-adjusted)\n",
|
||
"- **Close**: Last bar's close (ratio-adjusted)\n",
|
||
"- **Volume**: Sum of all volumes"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 13,
|
||
"id": "f6ea3cfe",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:32:56.328907Z",
|
||
"iopub.status.busy": "2026-06-13T02:32:56.328825Z",
|
||
"iopub.status.idle": "2026-06-13T02:32:56.704820Z",
|
||
"shell.execute_reply": "2026-06-13T02:32:56.704466Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.378456,
|
||
"end_time": "2026-06-13T02:32:56.705201+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:56.326745+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Daily bars: 312,859\n",
|
||
"Products: 30\n",
|
||
"Session date range: 2011-01-03 to 2025-12-31\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Aggregate to daily by session_date, product, tenor\n",
|
||
"daily = (\n",
|
||
" hourly_with_sessions.sort([\"product\", \"tenor\", \"timestamp\"])\n",
|
||
" .group_by([\"session_date\", \"product\", \"tenor\"])\n",
|
||
" .agg(\n",
|
||
" [\n",
|
||
" pl.col(\"open\").first(),\n",
|
||
" pl.col(\"high\").max(),\n",
|
||
" pl.col(\"low\").min(),\n",
|
||
" pl.col(\"close\").last(),\n",
|
||
" pl.col(\"volume\").sum(),\n",
|
||
" pl.len().alias(\"bar_count\"),\n",
|
||
" pl.col(\"timestamp\").min().alias(\"session_start\"),\n",
|
||
" pl.col(\"timestamp\").max().alias(\"session_end\"),\n",
|
||
" ]\n",
|
||
" )\n",
|
||
" .sort([\"product\", \"tenor\", \"session_date\"])\n",
|
||
")\n",
|
||
"\n",
|
||
"print(f\"Daily bars: {len(daily):,}\")\n",
|
||
"print(f\"Products: {daily['product'].n_unique()}\")\n",
|
||
"print(f\"Session date range: {daily['session_date'].min()} to {daily['session_date'].max()}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 14,
|
||
"id": "932ec406",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:32:56.709211Z",
|
||
"iopub.status.busy": "2026-06-13T02:32:56.709134Z",
|
||
"iopub.status.idle": "2026-06-13T02:32:56.712981Z",
|
||
"shell.execute_reply": "2026-06-13T02:32:56.712469Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.00646,
|
||
"end_time": "2026-06-13T02:32:56.713473+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:56.707013+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"ES front month daily bars (first 20 sessions):\n"
|
||
]
|
||
},
|
||
{
|
||
"data": {
|
||
"text/html": [
|
||
"<div><style>\n",
|
||
".dataframe > thead > tr,\n",
|
||
".dataframe > tbody > tr {\n",
|
||
" text-align: right;\n",
|
||
" white-space: pre-wrap;\n",
|
||
"}\n",
|
||
"</style>\n",
|
||
"<small>shape: (20, 7)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>session_date</th><th>open</th><th>high</th><th>low</th><th>close</th><th>volume</th><th>bar_count</th></tr><tr><td>date</td><td>f64</td><td>f64</td><td>f64</td><td>f64</td><td>u64</td><td>u32</td></tr></thead><tbody><tr><td>2011-01-03</td><td>1094.515418</td><td>1108.894005</td><td>1093.861846</td><td>1102.793998</td><td>1596140</td><td>23</td></tr><tr><td>2011-01-04</td><td>1102.793998</td><td>1106.715431</td><td>1096.476135</td><td>1102.793998</td><td>1764094</td><td>24</td></tr><tr><td>2011-01-05</td><td>1102.793998</td><td>1109.983291</td><td>1094.297561</td><td>1107.804718</td><td>1784553</td><td>24</td></tr><tr><td>2011-01-06</td><td>1107.804718</td><td>1112.815437</td><td>1103.44757</td><td>1106.497574</td><td>1597347</td><td>24</td></tr><tr><td>2011-01-07</td><td>1106.497574</td><td>1110.201149</td><td>1096.04042</td><td>1104.319</td><td>2166280</td><td>24</td></tr><tr><td>…</td><td>…</td><td>…</td><td>…</td><td>…</td><td>…</td><td>…</td></tr><tr><td>2011-01-24</td><td>1114.994011</td><td>1123.926163</td><td>1113.469009</td><td>1122.183304</td><td>1639436</td><td>23</td></tr><tr><td>2011-01-25</td><td>1122.183304</td><td>1125.669022</td><td>1112.815437</td><td>1121.529732</td><td>2359940</td><td>24</td></tr><tr><td>2011-01-26</td><td>1121.311875</td><td>1130.026169</td><td>1120.658303</td><td>1126.976166</td><td>1611687</td><td>24</td></tr><tr><td>2011-01-27</td><td>1126.976166</td><td>1131.333314</td><td>1123.490448</td><td>1128.501168</td><td>1549013</td><td>24</td></tr><tr><td>2011-01-28</td><td>1128.719025</td><td>1132.4226</td><td>1107.151146</td><td>1107.151146</td><td>3227539</td><td>24</td></tr></tbody></table></div>"
|
||
],
|
||
"text/plain": [
|
||
"shape: (20, 7)\n",
|
||
"┌──────────────┬─────────────┬─────────────┬─────────────┬─────────────┬─────────┬───────────┐\n",
|
||
"│ session_date ┆ open ┆ high ┆ low ┆ close ┆ volume ┆ bar_count │\n",
|
||
"│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │\n",
|
||
"│ date ┆ f64 ┆ f64 ┆ f64 ┆ f64 ┆ u64 ┆ u32 │\n",
|
||
"╞══════════════╪═════════════╪═════════════╪═════════════╪═════════════╪═════════╪═══════════╡\n",
|
||
"│ 2011-01-03 ┆ 1094.515418 ┆ 1108.894005 ┆ 1093.861846 ┆ 1102.793998 ┆ 1596140 ┆ 23 │\n",
|
||
"│ 2011-01-04 ┆ 1102.793998 ┆ 1106.715431 ┆ 1096.476135 ┆ 1102.793998 ┆ 1764094 ┆ 24 │\n",
|
||
"│ 2011-01-05 ┆ 1102.793998 ┆ 1109.983291 ┆ 1094.297561 ┆ 1107.804718 ┆ 1784553 ┆ 24 │\n",
|
||
"│ 2011-01-06 ┆ 1107.804718 ┆ 1112.815437 ┆ 1103.44757 ┆ 1106.497574 ┆ 1597347 ┆ 24 │\n",
|
||
"│ 2011-01-07 ┆ 1106.497574 ┆ 1110.201149 ┆ 1096.04042 ┆ 1104.319 ┆ 2166280 ┆ 24 │\n",
|
||
"│ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … │\n",
|
||
"│ 2011-01-24 ┆ 1114.994011 ┆ 1123.926163 ┆ 1113.469009 ┆ 1122.183304 ┆ 1639436 ┆ 23 │\n",
|
||
"│ 2011-01-25 ┆ 1122.183304 ┆ 1125.669022 ┆ 1112.815437 ┆ 1121.529732 ┆ 2359940 ┆ 24 │\n",
|
||
"│ 2011-01-26 ┆ 1121.311875 ┆ 1130.026169 ┆ 1120.658303 ┆ 1126.976166 ┆ 1611687 ┆ 24 │\n",
|
||
"│ 2011-01-27 ┆ 1126.976166 ┆ 1131.333314 ┆ 1123.490448 ┆ 1128.501168 ┆ 1549013 ┆ 24 │\n",
|
||
"│ 2011-01-28 ┆ 1128.719025 ┆ 1132.4226 ┆ 1107.151146 ┆ 1107.151146 ┆ 3227539 ┆ 24 │\n",
|
||
"└──────────────┴─────────────┴─────────────┴─────────────┴─────────────┴─────────┴───────────┘"
|
||
]
|
||
},
|
||
"execution_count": 14,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"es_daily = daily.filter((pl.col(\"product\") == \"ES\") & (pl.col(\"tenor\") == 0))\n",
|
||
"print(\"ES front month daily bars (first 20 sessions):\")\n",
|
||
"es_daily.select(\"session_date\", \"open\", \"high\", \"low\", \"close\", \"volume\", \"bar_count\").head(20)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "f9c8cc98",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.00181,
|
||
"end_time": "2026-06-13T02:32:56.717345+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:56.715535+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"## 5. Validate Aggregation\n",
|
||
"\n",
|
||
"Check that daily aggregation is correct:\n",
|
||
"- Bar counts should be ~23 per session (23-hour trading day)\n",
|
||
"- OHLC relationships should hold (Low ≤ Open/Close ≤ High)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 15,
|
||
"id": "a6b6418c",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:32:56.721349Z",
|
||
"iopub.status.busy": "2026-06-13T02:32:56.721207Z",
|
||
"iopub.status.idle": "2026-06-13T02:32:56.728721Z",
|
||
"shell.execute_reply": "2026-06-13T02:32:56.728384Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.010175,
|
||
"end_time": "2026-06-13T02:32:56.729191+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:56.719016+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Typical sessions (20-24 bars): 150,090 / 312,859\n",
|
||
"Bar counts per session:\n"
|
||
]
|
||
},
|
||
{
|
||
"data": {
|
||
"text/html": [
|
||
"<div><style>\n",
|
||
".dataframe > thead > tr,\n",
|
||
".dataframe > tbody > tr {\n",
|
||
" text-align: right;\n",
|
||
" white-space: pre-wrap;\n",
|
||
"}\n",
|
||
"</style>\n",
|
||
"<small>shape: (25, 2)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>bar_count</th><th>len</th></tr><tr><td>u32</td><td>u32</td></tr></thead><tbody><tr><td>1</td><td>6922</td></tr><tr><td>2</td><td>4704</td></tr><tr><td>3</td><td>3990</td></tr><tr><td>4</td><td>3771</td></tr><tr><td>5</td><td>3986</td></tr><tr><td>…</td><td>…</td></tr><tr><td>21</td><td>11408</td></tr><tr><td>22</td><td>11093</td></tr><tr><td>23</td><td>106441</td></tr><tr><td>24</td><td>13625</td></tr><tr><td>25</td><td>3540</td></tr></tbody></table></div>"
|
||
],
|
||
"text/plain": [
|
||
"shape: (25, 2)\n",
|
||
"┌───────────┬────────┐\n",
|
||
"│ bar_count ┆ len │\n",
|
||
"│ --- ┆ --- │\n",
|
||
"│ u32 ┆ u32 │\n",
|
||
"╞═══════════╪════════╡\n",
|
||
"│ 1 ┆ 6922 │\n",
|
||
"│ 2 ┆ 4704 │\n",
|
||
"│ 3 ┆ 3990 │\n",
|
||
"│ 4 ┆ 3771 │\n",
|
||
"│ 5 ┆ 3986 │\n",
|
||
"│ … ┆ … │\n",
|
||
"│ 21 ┆ 11408 │\n",
|
||
"│ 22 ┆ 11093 │\n",
|
||
"│ 23 ┆ 106441 │\n",
|
||
"│ 24 ┆ 13625 │\n",
|
||
"│ 25 ┆ 3540 │\n",
|
||
"└───────────┴────────┘"
|
||
]
|
||
},
|
||
"execution_count": 15,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"bar_counts = daily.group_by(\"bar_count\").len().sort(\"bar_count\")\n",
|
||
"typical_sessions = daily.filter(pl.col(\"bar_count\").is_between(20, 24))\n",
|
||
"print(f\"Typical sessions (20-24 bars): {len(typical_sessions):,} / {len(daily):,}\")\n",
|
||
"print(\"Bar counts per session:\")\n",
|
||
"bar_counts"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 16,
|
||
"id": "34438939",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:32:56.735884Z",
|
||
"iopub.status.busy": "2026-06-13T02:32:56.735775Z",
|
||
"iopub.status.idle": "2026-06-13T02:32:56.740634Z",
|
||
"shell.execute_reply": "2026-06-13T02:32:56.740259Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.00816,
|
||
"end_time": "2026-06-13T02:32:56.740946+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:56.732786+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"OHLC Invariant Check:\n",
|
||
" [OK] low_le_open: 100.00%\n",
|
||
" [OK] low_le_close: 100.00%\n",
|
||
" [OK] high_ge_open: 100.00%\n",
|
||
" [OK] high_ge_close: 100.00%\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# OHLC invariant check\n",
|
||
"ohlc_check = daily.with_columns(\n",
|
||
" [\n",
|
||
" (pl.col(\"low\") <= pl.col(\"open\")).alias(\"low_le_open\"),\n",
|
||
" (pl.col(\"low\") <= pl.col(\"close\")).alias(\"low_le_close\"),\n",
|
||
" (pl.col(\"high\") >= pl.col(\"open\")).alias(\"high_ge_open\"),\n",
|
||
" (pl.col(\"high\") >= pl.col(\"close\")).alias(\"high_ge_close\"),\n",
|
||
" ]\n",
|
||
")\n",
|
||
"\n",
|
||
"print(\"OHLC Invariant Check:\")\n",
|
||
"for col in [\"low_le_open\", \"low_le_close\", \"high_ge_open\", \"high_ge_close\"]:\n",
|
||
" pct = ohlc_check[col].mean() * 100\n",
|
||
" status = \"[OK]\" if pct > 99.9 else \"[FAIL]\"\n",
|
||
" print(f\" {status} {col}: {pct:.2f}%\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "2ec3c357",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.001867,
|
||
"end_time": "2026-06-13T02:32:56.744732+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:56.742865+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"## 6. Coverage Summary\n",
|
||
"\n",
|
||
"Summary of daily data coverage by product."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 17,
|
||
"id": "a8bae8ac",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:32:56.748704Z",
|
||
"iopub.status.busy": "2026-06-13T02:32:56.748617Z",
|
||
"iopub.status.idle": "2026-06-13T02:32:56.757236Z",
|
||
"shell.execute_reply": "2026-06-13T02:32:56.756933Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.011292,
|
||
"end_time": "2026-06-13T02:32:56.757756+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:56.746464+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Daily data coverage by product:\n"
|
||
]
|
||
},
|
||
{
|
||
"data": {
|
||
"text/html": [
|
||
"<div><style>\n",
|
||
".dataframe > thead > tr,\n",
|
||
".dataframe > tbody > tr {\n",
|
||
" text-align: right;\n",
|
||
" white-space: pre-wrap;\n",
|
||
"}\n",
|
||
"</style>\n",
|
||
"<small>shape: (30, 5)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>product</th><th>start_date</th><th>end_date</th><th>total_bars</th><th>tenors</th></tr><tr><td>str</td><td>date</td><td>date</td><td>u32</td><td>u32</td></tr></thead><tbody><tr><td>"6A"</td><td>2011-01-03</td><td>2025-12-31</td><td>10499</td><td>3</td></tr><tr><td>"6B"</td><td>2011-01-03</td><td>2025-12-31</td><td>10393</td><td>3</td></tr><tr><td>"6C"</td><td>2011-01-03</td><td>2025-12-31</td><td>11249</td><td>3</td></tr><tr><td>"6E"</td><td>2011-01-03</td><td>2025-12-31</td><td>11322</td><td>3</td></tr><tr><td>"6J"</td><td>2011-01-03</td><td>2025-12-31</td><td>10819</td><td>3</td></tr><tr><td>…</td><td>…</td><td>…</td><td>…</td><td>…</td></tr><tr><td>"ZM"</td><td>2011-01-03</td><td>2025-12-30</td><td>11319</td><td>3</td></tr><tr><td>"ZN"</td><td>2011-01-03</td><td>2025-12-31</td><td>8305</td><td>3</td></tr><tr><td>"ZS"</td><td>2011-01-03</td><td>2025-12-30</td><td>11320</td><td>3</td></tr><tr><td>"ZT"</td><td>2011-01-03</td><td>2025-12-31</td><td>7362</td><td>3</td></tr><tr><td>"ZW"</td><td>2011-01-03</td><td>2025-12-30</td><td>11320</td><td>3</td></tr></tbody></table></div>"
|
||
],
|
||
"text/plain": [
|
||
"shape: (30, 5)\n",
|
||
"┌─────────┬────────────┬────────────┬────────────┬────────┐\n",
|
||
"│ product ┆ start_date ┆ end_date ┆ total_bars ┆ tenors │\n",
|
||
"│ --- ┆ --- ┆ --- ┆ --- ┆ --- │\n",
|
||
"│ str ┆ date ┆ date ┆ u32 ┆ u32 │\n",
|
||
"╞═════════╪════════════╪════════════╪════════════╪════════╡\n",
|
||
"│ 6A ┆ 2011-01-03 ┆ 2025-12-31 ┆ 10499 ┆ 3 │\n",
|
||
"│ 6B ┆ 2011-01-03 ┆ 2025-12-31 ┆ 10393 ┆ 3 │\n",
|
||
"│ 6C ┆ 2011-01-03 ┆ 2025-12-31 ┆ 11249 ┆ 3 │\n",
|
||
"│ 6E ┆ 2011-01-03 ┆ 2025-12-31 ┆ 11322 ┆ 3 │\n",
|
||
"│ 6J ┆ 2011-01-03 ┆ 2025-12-31 ┆ 10819 ┆ 3 │\n",
|
||
"│ … ┆ … ┆ … ┆ … ┆ … │\n",
|
||
"│ ZM ┆ 2011-01-03 ┆ 2025-12-30 ┆ 11319 ┆ 3 │\n",
|
||
"│ ZN ┆ 2011-01-03 ┆ 2025-12-31 ┆ 8305 ┆ 3 │\n",
|
||
"│ ZS ┆ 2011-01-03 ┆ 2025-12-30 ┆ 11320 ┆ 3 │\n",
|
||
"│ ZT ┆ 2011-01-03 ┆ 2025-12-31 ┆ 7362 ┆ 3 │\n",
|
||
"│ ZW ┆ 2011-01-03 ┆ 2025-12-30 ┆ 11320 ┆ 3 │\n",
|
||
"└─────────┴────────────┴────────────┴────────────┴────────┘"
|
||
]
|
||
},
|
||
"execution_count": 17,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"# Coverage by product\n",
|
||
"coverage = (\n",
|
||
" daily.group_by(\"product\")\n",
|
||
" .agg(\n",
|
||
" [\n",
|
||
" pl.col(\"session_date\").min().alias(\"start_date\"),\n",
|
||
" pl.col(\"session_date\").max().alias(\"end_date\"),\n",
|
||
" pl.len().alias(\"total_bars\"),\n",
|
||
" pl.col(\"tenor\").n_unique().alias(\"tenors\"),\n",
|
||
" ]\n",
|
||
" )\n",
|
||
" .sort(\"product\")\n",
|
||
")\n",
|
||
"\n",
|
||
"print(\"Daily data coverage by product:\")\n",
|
||
"coverage"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 18,
|
||
"id": "973556b5",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:32:56.762495Z",
|
||
"iopub.status.busy": "2026-06-13T02:32:56.762410Z",
|
||
"iopub.status.idle": "2026-06-13T02:32:56.770238Z",
|
||
"shell.execute_reply": "2026-06-13T02:32:56.769959Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.010728,
|
||
"end_time": "2026-06-13T02:32:56.770567+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:56.759839+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Coverage by tenor:\n"
|
||
]
|
||
},
|
||
{
|
||
"data": {
|
||
"text/html": [
|
||
"<div><style>\n",
|
||
".dataframe > thead > tr,\n",
|
||
".dataframe > tbody > tr {\n",
|
||
" text-align: right;\n",
|
||
" white-space: pre-wrap;\n",
|
||
"}\n",
|
||
"</style>\n",
|
||
"<small>shape: (3, 3)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>tenor</th><th>products</th><th>total_bars</th></tr><tr><td>i8</td><td>u32</td><td>u32</td></tr></thead><tbody><tr><td>0</td><td>30</td><td>113506</td></tr><tr><td>1</td><td>30</td><td>111441</td></tr><tr><td>2</td><td>30</td><td>87912</td></tr></tbody></table></div>"
|
||
],
|
||
"text/plain": [
|
||
"shape: (3, 3)\n",
|
||
"┌───────┬──────────┬────────────┐\n",
|
||
"│ tenor ┆ products ┆ total_bars │\n",
|
||
"│ --- ┆ --- ┆ --- │\n",
|
||
"│ i8 ┆ u32 ┆ u32 │\n",
|
||
"╞═══════╪══════════╪════════════╡\n",
|
||
"│ 0 ┆ 30 ┆ 113506 │\n",
|
||
"│ 1 ┆ 30 ┆ 111441 │\n",
|
||
"│ 2 ┆ 30 ┆ 87912 │\n",
|
||
"└───────┴──────────┴────────────┘"
|
||
]
|
||
},
|
||
"execution_count": 18,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"tenor_coverage = (\n",
|
||
" daily.group_by(\"tenor\")\n",
|
||
" .agg(\n",
|
||
" pl.col(\"product\").n_unique().alias(\"products\"),\n",
|
||
" pl.len().alias(\"total_bars\"),\n",
|
||
" )\n",
|
||
" .sort(\"tenor\")\n",
|
||
")\n",
|
||
"print(\"Coverage by tenor:\")\n",
|
||
"tenor_coverage"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "0d116259",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.001815,
|
||
"end_time": "2026-06-13T02:32:56.774504+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:56.772689+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"## 7. Save Daily Data\n",
|
||
"\n",
|
||
"Save the session-aggregated daily data for downstream use."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 19,
|
||
"id": "24e2b917",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:32:56.778625Z",
|
||
"iopub.status.busy": "2026-06-13T02:32:56.778536Z",
|
||
"iopub.status.idle": "2026-06-13T02:32:56.807111Z",
|
||
"shell.execute_reply": "2026-06-13T02:32:56.806221Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.03117,
|
||
"end_time": "2026-06-13T02:32:56.807469+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:56.776299+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Saved: 02_financial_data_universe/output/futures_daily/continuous_daily.parquet\n",
|
||
"Size: 11.7 MB\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Create output directory\n",
|
||
"OUTPUT_DIR.mkdir(parents=True, exist_ok=True)\n",
|
||
"\n",
|
||
"# Save combined daily file\n",
|
||
"output_path = OUTPUT_DIR / \"continuous_daily.parquet\"\n",
|
||
"daily.write_parquet(output_path)\n",
|
||
"print(f\"Saved: {output_path}\")\n",
|
||
"print(f\"Size: {output_path.stat().st_size / 1e6:.1f} MB\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 20,
|
||
"id": "b5dfc5af",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:32:56.812171Z",
|
||
"iopub.status.busy": "2026-06-13T02:32:56.812051Z",
|
||
"iopub.status.idle": "2026-06-13T02:32:56.935274Z",
|
||
"shell.execute_reply": "2026-06-13T02:32:56.934551Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.127291,
|
||
"end_time": "2026-06-13T02:32:56.936788+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:56.809497+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"\n",
|
||
"Saved per-product files to: 02_financial_data_universe/output/futures_daily/by_product/\n",
|
||
"Products: 30\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Also save per-product files for convenience\n",
|
||
"per_product_dir = OUTPUT_DIR / \"by_product\"\n",
|
||
"per_product_dir.mkdir(exist_ok=True)\n",
|
||
"\n",
|
||
"for product in products:\n",
|
||
" product_df = daily.filter(pl.col(\"product\") == product)\n",
|
||
" product_path = per_product_dir / f\"{product}.parquet\"\n",
|
||
" product_df.write_parquet(product_path)\n",
|
||
"\n",
|
||
"print(f\"\\nSaved per-product files to: {per_product_dir}/\")\n",
|
||
"print(f\"Products: {len(products)}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "14834590",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.003529,
|
||
"end_time": "2026-06-13T02:32:56.946706+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:56.943177+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"## 8. Using the Daily Data\n",
|
||
"\n",
|
||
"The daily data is now available via `load_cme_futures()` (daily is the default frequency).\n",
|
||
"This loader is defined in `data/__init__.py` and can be used by downstream chapters."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 21,
|
||
"id": "4a3e2cf3",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:32:56.951587Z",
|
||
"iopub.status.busy": "2026-06-13T02:32:56.951498Z",
|
||
"iopub.status.idle": "2026-06-13T02:32:56.966807Z",
|
||
"shell.execute_reply": "2026-06-13T02:32:56.966535Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.018133,
|
||
"end_time": "2026-06-13T02:32:56.967259+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:56.949126+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"ES + NQ front month, 2024: 518 daily bars\n"
|
||
]
|
||
},
|
||
{
|
||
"data": {
|
||
"text/html": [
|
||
"<div><style>\n",
|
||
".dataframe > thead > tr,\n",
|
||
".dataframe > tbody > tr {\n",
|
||
" text-align: right;\n",
|
||
" white-space: pre-wrap;\n",
|
||
"}\n",
|
||
"</style>\n",
|
||
"<small>shape: (10, 11)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>session_date</th><th>product</th><th>tenor</th><th>open</th><th>high</th><th>low</th><th>close</th><th>volume</th><th>bar_count</th><th>session_start</th><th>session_end</th></tr><tr><td>date</td><td>str</td><td>i8</td><td>f64</td><td>f64</td><td>f64</td><td>f64</td><td>u64</td><td>u32</td><td>datetime[ns, UTC]</td><td>datetime[ns, UTC]</td></tr></thead><tbody><tr><td>2024-01-02</td><td>"ES"</td><td>0</td><td>5344.188616</td><td>5355.280747</td><td>5285.954929</td><td>5311.46683</td><td>1549005</td><td>23</td><td>2024-01-01 23:00:00 UTC</td><td>2024-01-02 21:00:00 UTC</td></tr><tr><td>2024-01-02</td><td>"NQ"</td><td>0</td><td>18663.756856</td><td>18685.141383</td><td>18228.938148</td><td>18348.746329</td><td>634193</td><td>23</td><td>2024-01-01 23:00:00 UTC</td><td>2024-01-02 21:00:00 UTC</td></tr><tr><td>2024-01-03</td><td>"ES"</td><td>0</td><td>5313.96256</td><td>5313.96256</td><td>5258.779209</td><td>5267.37561</td><td>1587336</td><td>23</td><td>2024-01-02 23:00:00 UTC</td><td>2024-01-03 21:00:00 UTC</td></tr><tr><td>2024-01-03</td><td>"NQ"</td><td>0</td><td>18354.229541</td><td>18354.777862</td><td>18118.725588</td><td>18141.480917</td><td>668346</td><td>23</td><td>2024-01-02 23:00:00 UTC</td><td>2024-01-03 21:00:00 UTC</td></tr><tr><td>2024-01-04</td><td>"ES"</td><td>0</td><td>5268.762127</td><td>5287.064142</td><td>5243.250226</td><td>5249.073594</td><td>1299377</td><td>23</td><td>2024-01-03 23:00:00 UTC</td><td>2024-01-04 21:00:00 UTC</td></tr><tr><td>2024-01-04</td><td>"NQ"</td><td>0</td><td>18144.222523</td><td>18190.281504</td><td>18024.414342</td><td>18040.589817</td><td>551255</td><td>23</td><td>2024-01-03 23:00:00 UTC</td><td>2024-01-04 21:00:00 UTC</td></tr><tr><td>2024-01-05</td><td>"ES"</td><td>0</td><td>5249.628201</td><td>5280.131561</td><td>5215.519899</td><td>5252.678537</td><td>1663860</td><td>23</td><td>2024-01-04 23:00:00 UTC</td><td>2024-01-05 21:00:00 UTC</td></tr><tr><td>2024-01-05</td><td>"NQ"</td><td>0</td><td>18045.250548</td><td>18179.31508</td><td>17912.830979</td><td>18056.216971</td><td>620632</td><td>23</td><td>2024-01-04 23:00:00 UTC</td><td>2024-01-05 21:00:00 UTC</td></tr><tr><td>2024-01-08</td><td>"ES"</td><td>0</td><td>5252.95584</td><td>5327.827723</td><td>5230.216972</td><td>5322.004354</td><td>1339989</td><td>23</td><td>2024-01-07 23:00:00 UTC</td><td>2024-01-08 21:00:00 UTC</td></tr><tr><td>2024-01-08</td><td>"NQ"</td><td>0</td><td>18061.700183</td><td>18435.929399</td><td>17961.083244</td><td>18410.706624</td><td>537622</td><td>23</td><td>2024-01-07 23:00:00 UTC</td><td>2024-01-08 21:00:00 UTC</td></tr></tbody></table></div>"
|
||
],
|
||
"text/plain": [
|
||
"shape: (10, 11)\n",
|
||
"┌─────────────┬─────────┬───────┬─────────────┬───┬─────────┬───────────┬─────────────┬────────────┐\n",
|
||
"│ session_dat ┆ product ┆ tenor ┆ open ┆ … ┆ volume ┆ bar_count ┆ session_sta ┆ session_en │\n",
|
||
"│ e ┆ --- ┆ --- ┆ --- ┆ ┆ --- ┆ --- ┆ rt ┆ d │\n",
|
||
"│ --- ┆ str ┆ i8 ┆ f64 ┆ ┆ u64 ┆ u32 ┆ --- ┆ --- │\n",
|
||
"│ date ┆ ┆ ┆ ┆ ┆ ┆ ┆ datetime[ns ┆ datetime[n │\n",
|
||
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ , UTC] ┆ s, UTC] │\n",
|
||
"╞═════════════╪═════════╪═══════╪═════════════╪═══╪═════════╪═══════════╪═════════════╪════════════╡\n",
|
||
"│ 2024-01-02 ┆ ES ┆ 0 ┆ 5344.188616 ┆ … ┆ 1549005 ┆ 23 ┆ 2024-01-01 ┆ 2024-01-02 │\n",
|
||
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ 23:00:00 ┆ 21:00:00 │\n",
|
||
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ UTC ┆ UTC │\n",
|
||
"│ 2024-01-02 ┆ NQ ┆ 0 ┆ 18663.75685 ┆ … ┆ 634193 ┆ 23 ┆ 2024-01-01 ┆ 2024-01-02 │\n",
|
||
"│ ┆ ┆ ┆ 6 ┆ ┆ ┆ ┆ 23:00:00 ┆ 21:00:00 │\n",
|
||
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ UTC ┆ UTC │\n",
|
||
"│ 2024-01-03 ┆ ES ┆ 0 ┆ 5313.96256 ┆ … ┆ 1587336 ┆ 23 ┆ 2024-01-02 ┆ 2024-01-03 │\n",
|
||
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ 23:00:00 ┆ 21:00:00 │\n",
|
||
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ UTC ┆ UTC │\n",
|
||
"│ 2024-01-03 ┆ NQ ┆ 0 ┆ 18354.22954 ┆ … ┆ 668346 ┆ 23 ┆ 2024-01-02 ┆ 2024-01-03 │\n",
|
||
"│ ┆ ┆ ┆ 1 ┆ ┆ ┆ ┆ 23:00:00 ┆ 21:00:00 │\n",
|
||
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ UTC ┆ UTC │\n",
|
||
"│ 2024-01-04 ┆ ES ┆ 0 ┆ 5268.762127 ┆ … ┆ 1299377 ┆ 23 ┆ 2024-01-03 ┆ 2024-01-04 │\n",
|
||
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ 23:00:00 ┆ 21:00:00 │\n",
|
||
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ UTC ┆ UTC │\n",
|
||
"│ 2024-01-04 ┆ NQ ┆ 0 ┆ 18144.22252 ┆ … ┆ 551255 ┆ 23 ┆ 2024-01-03 ┆ 2024-01-04 │\n",
|
||
"│ ┆ ┆ ┆ 3 ┆ ┆ ┆ ┆ 23:00:00 ┆ 21:00:00 │\n",
|
||
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ UTC ┆ UTC │\n",
|
||
"│ 2024-01-05 ┆ ES ┆ 0 ┆ 5249.628201 ┆ … ┆ 1663860 ┆ 23 ┆ 2024-01-04 ┆ 2024-01-05 │\n",
|
||
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ 23:00:00 ┆ 21:00:00 │\n",
|
||
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ UTC ┆ UTC │\n",
|
||
"│ 2024-01-05 ┆ NQ ┆ 0 ┆ 18045.25054 ┆ … ┆ 620632 ┆ 23 ┆ 2024-01-04 ┆ 2024-01-05 │\n",
|
||
"│ ┆ ┆ ┆ 8 ┆ ┆ ┆ ┆ 23:00:00 ┆ 21:00:00 │\n",
|
||
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ UTC ┆ UTC │\n",
|
||
"│ 2024-01-08 ┆ ES ┆ 0 ┆ 5252.95584 ┆ … ┆ 1339989 ┆ 23 ┆ 2024-01-07 ┆ 2024-01-08 │\n",
|
||
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ 23:00:00 ┆ 21:00:00 │\n",
|
||
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ UTC ┆ UTC │\n",
|
||
"│ 2024-01-08 ┆ NQ ┆ 0 ┆ 18061.70018 ┆ … ┆ 537622 ┆ 23 ┆ 2024-01-07 ┆ 2024-01-08 │\n",
|
||
"│ ┆ ┆ ┆ 3 ┆ ┆ ┆ ┆ 23:00:00 ┆ 21:00:00 │\n",
|
||
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ UTC ┆ UTC │\n",
|
||
"└─────────────┴─────────┴───────┴─────────────┴───┴─────────┴───────────┴─────────────┴────────────┘"
|
||
]
|
||
},
|
||
"execution_count": 21,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"es_nq_2024 = (\n",
|
||
" pl.read_parquet(OUTPUT_DIR / \"continuous_daily.parquet\")\n",
|
||
" .filter(\n",
|
||
" pl.col(\"product\").is_in([\"ES\", \"NQ\"])\n",
|
||
" & (pl.col(\"tenor\") == 0)\n",
|
||
" & (pl.col(\"session_date\") >= pl.lit(\"2024-01-01\").str.to_date())\n",
|
||
" & (pl.col(\"session_date\") <= pl.lit(\"2024-12-31\").str.to_date())\n",
|
||
" )\n",
|
||
" .sort(\"session_date\", \"product\")\n",
|
||
")\n",
|
||
"print(f\"ES + NQ front month, 2024: {len(es_nq_2024)} daily bars\")\n",
|
||
"es_nq_2024.head(10)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "ae0cb38a",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.002177,
|
||
"end_time": "2026-06-13T02:32:56.972050+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:32:56.969873+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"## Key Takeaways\n",
|
||
"\n",
|
||
"1. **CME sessions end at 4 PM CT**, not midnight UTC. The session date is\n",
|
||
" the date the session ends — Sunday-evening trading belongs to Monday's\n",
|
||
" session.\n",
|
||
"2. **Volume here is 5,463,741 hourly bars across 30 products and 3 tenors**,\n",
|
||
" aggregating to 312,859 daily bars over 2011-01-03 through 2025-12-31.\n",
|
||
"3. **Ratio back-adjustment** is applied per (product, tenor) before\n",
|
||
" aggregation — for ES front month the cumulative ratio ranges 0.87–1.16\n",
|
||
" over 427 rolls, preserving percentage returns across roll boundaries.\n",
|
||
"4. **Full sessions have 23 hourly bars** (23-hour trading day): the 23-bar\n",
|
||
" bucket is by far the largest in the bar-count distribution. Shorter\n",
|
||
" sessions arise from holidays, deferred tenors with thin trading, and\n",
|
||
" partial days.\n",
|
||
"5. **OHLC invariants hold at 100%** on the aggregated daily bars across all\n",
|
||
" four checks.\n",
|
||
"\n",
|
||
"## Next Steps\n",
|
||
"\n",
|
||
"- [`06_futures_continuous`](06_futures_continuous.ipynb): Roll detection and\n",
|
||
" alternative adjustment methods (Panama / calendar).\n",
|
||
"- **Chapter 8**: Feature engineering on daily futures data.\n",
|
||
"- **Chapter 16**: Backtesting with session-correct returns."
|
||
]
|
||
}
|
||
],
|
||
"metadata": {
|
||
"kernelspec": {
|
||
"display_name": "Python 3 (ipykernel)",
|
||
"language": "python",
|
||
"name": "python3"
|
||
},
|
||
"language_info": {
|
||
"codemirror_mode": {
|
||
"name": "ipython",
|
||
"version": 3
|
||
},
|
||
"file_extension": ".py",
|
||
"mimetype": "text/x-python",
|
||
"name": "python",
|
||
"nbconvert_exporter": "python",
|
||
"pygments_lexer": "ipython3",
|
||
"version": "3.14.3"
|
||
},
|
||
"papermill": {
|
||
"default_parameters": {},
|
||
"duration": 10.155364,
|
||
"end_time": "2026-06-13T02:32:57.892438+00:00",
|
||
"environment_variables": {},
|
||
"exception": null,
|
||
"input_path": "02_financial_data_universe/05_futures_session_aggregation.ipynb",
|
||
"output_path": "02_financial_data_universe/05_futures_session_aggregation.ipynb",
|
||
"parameters": {},
|
||
"start_time": "2026-06-13T02:32:47.737074+00:00",
|
||
"version": "2.7.0"
|
||
},
|
||
"jupytext": {
|
||
"cell_metadata_filter": "tags,-all"
|
||
},
|
||
"ml4t_provenance": {
|
||
"source_py_blob": "d264231a61ef1799abe8e09b447b13f93e47985a",
|
||
"executed_at": "2026-06-13T02:32:58.113199+00:00",
|
||
"executor": "cpu-local",
|
||
"production": true,
|
||
"parameters": {},
|
||
"notes": "executed-state finalization batch (2026-06-13 run)"
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 5
|
||
}
|