{
"cells": [
{
"cell_type": "markdown",
"id": "6f72361a",
"metadata": {
"papermill": {
"duration": 0.001744,
"end_time": "2026-06-13T03:31:19.217144+00:00",
"exception": false,
"start_time": "2026-06-13T03:31:19.215400+00:00",
"status": "completed"
}
},
"source": [
"# Microstructure Features\n",
"\n",
"**Chapter 8: Feature Engineering**\n",
"**Section Reference**: 8.2 - Price-Derived Features (Microstructure)\n",
"**Docker image**: `ml4t`\n",
"\n",
"## Purpose\n",
"\n",
"Microstructure features capture market dynamics invisible in daily OHLCV data.\n",
"They proxy for **liquidity**, **information flow**, and **execution quality**.\n",
"\n",
"## Learning Objectives\n",
"\n",
"1. Compute trade-based liquidity features (Kyle λ, Amihud, Roll spread)\n",
"2. Understand order flow imbalance and its predictive content\n",
"3. Distinguish between **flow features** and **state features** (critical!)\n",
"4. Know which features are alpha vs feasibility/cost inputs\n",
"\n",
"## Feature Categories\n",
"\n",
"| Category | Features | Data Required |\n",
"|----------|----------|---------------|\n",
"| **Liquidity** | Kyle λ, Amihud, Roll | OHLCV bars |\n",
"| **Order Flow** | OFI, trade intensity | Trade data |\n",
"| **Book State** | Spread, depth | LOB snapshots |\n",
"\n",
"## Data Policy\n",
"\n",
"Uses **real NASDAQ ITCH data**. The notebook raises a clear error when ITCH\n",
"is missing rather than substituting a synthetic toy panel."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "8e2ec790",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T03:31:19.221252Z",
"iopub.status.busy": "2026-06-13T03:31:19.221081Z",
"iopub.status.idle": "2026-06-13T03:31:21.446495Z",
"shell.execute_reply": "2026-06-13T03:31:21.445744Z"
},
"papermill": {
"duration": 2.228306,
"end_time": "2026-06-13T03:31:21.446924+00:00",
"exception": false,
"start_time": "2026-06-13T03:31:19.218618+00:00",
"status": "completed"
}
},
"outputs": [],
"source": [
"\"\"\"Microstructure Features — compute trade-based liquidity and order flow features from tick data.\"\"\"\n",
"\n",
"from __future__ import annotations\n",
"\n",
"import warnings\n",
"from datetime import time\n",
"\n",
"import plotly.graph_objects as go\n",
"import polars as pl\n",
"from plotly.subplots import make_subplots\n",
"\n",
"from utils.reproducibility import set_global_seeds\n",
"\n",
"warnings.filterwarnings(\"ignore\")"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "81cd4a24",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T03:31:21.460542Z",
"iopub.status.busy": "2026-06-13T03:31:21.460380Z",
"iopub.status.idle": "2026-06-13T03:31:21.462407Z",
"shell.execute_reply": "2026-06-13T03:31:21.462067Z"
},
"papermill": {
"duration": 0.014494,
"end_time": "2026-06-13T03:31:21.462795+00:00",
"exception": false,
"start_time": "2026-06-13T03:31:21.448301+00:00",
"status": "completed"
},
"tags": [
"parameters"
]
},
"outputs": [],
"source": [
"SEED = 42"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "d0f10d26",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T03:31:21.466080Z",
"iopub.status.busy": "2026-06-13T03:31:21.465955Z",
"iopub.status.idle": "2026-06-13T03:31:21.505356Z",
"shell.execute_reply": "2026-06-13T03:31:21.504775Z"
},
"lines_to_next_cell": 2,
"papermill": {
"duration": 0.041765,
"end_time": "2026-06-13T03:31:21.505837+00:00",
"exception": false,
"start_time": "2026-06-13T03:31:21.464072+00:00",
"status": "completed"
}
},
"outputs": [],
"source": [
"set_global_seeds(SEED)"
]
},
{
"cell_type": "markdown",
"id": "8aef3441",
"metadata": {
"papermill": {
"duration": 0.001187,
"end_time": "2026-06-13T03:31:21.508375+00:00",
"exception": false,
"start_time": "2026-06-13T03:31:21.507188+00:00",
"status": "completed"
}
},
"source": [
"## 1. Data Loading with Availability Check\n",
"\n",
"Microstructure analysis requires high-frequency data. The loader raises\n",
"a clear error if ITCH is missing — no silent fallback to synthetic data."
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "5cc633dc",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T03:31:21.511483Z",
"iopub.status.busy": "2026-06-13T03:31:21.511276Z",
"iopub.status.idle": "2026-06-13T03:31:21.557058Z",
"shell.execute_reply": "2026-06-13T03:31:21.556434Z"
},
"papermill": {
"duration": 0.047772,
"end_time": "2026-06-13T03:31:21.557332+00:00",
"exception": false,
"start_time": "2026-06-13T03:31:21.509560+00:00",
"status": "completed"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"ITCH data available: 15,114 trade messages loaded\n"
]
}
],
"source": [
"# Load real ITCH trade data — the notebook fails loudly if the data is\n",
"# missing rather than silently substituting a synthetic toy panel.\n",
"\n",
"from data import load_nasdaq_itch\n",
"\n",
"sample = load_nasdaq_itch(message_types=[\"P\"], symbols=[\"AAPL\"])\n",
"if len(sample) < 100:\n",
" raise RuntimeError(\n",
" f\"Expected NASDAQ ITCH trade data with >=100 rows for AAPL; got {len(sample)}. \"\n",
" \"Ensure ML4T_DATA_PATH is set and the ITCH dataset is downloaded.\"\n",
" )\n",
"print(f\"ITCH data available: {len(sample):,} trade messages loaded\")"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "65d09a03",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T03:31:21.560741Z",
"iopub.status.busy": "2026-06-13T03:31:21.560648Z",
"iopub.status.idle": "2026-06-13T03:31:21.600015Z",
"shell.execute_reply": "2026-06-13T03:31:21.599450Z"
},
"papermill": {
"duration": 0.041582,
"end_time": "2026-06-13T03:31:21.600381+00:00",
"exception": false,
"start_time": "2026-06-13T03:31:21.558799+00:00",
"status": "completed"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Loaded 88,572 trades across 3 stocks\n"
]
}
],
"source": [
"trades = load_nasdaq_itch(message_types=[\"P\"], symbols=[\"AAPL\", \"MSFT\", \"TSLA\"])\n",
"\n",
"# Convert price from 10,000ths to dollars\n",
"trades = trades.with_columns((pl.col(\"price\") / 10000.0).alias(\"price\"))\n",
"\n",
"# Filter to regular trading hours\n",
"trades = trades.filter(\n",
" (pl.col(\"timestamp\").dt.time() >= time(9, 30)) & (pl.col(\"timestamp\").dt.time() < time(16, 0))\n",
").sort([\"stock\", \"timestamp\"])\n",
"\n",
"print(f\"Loaded {len(trades):,} trades across {trades['stock'].n_unique()} stocks\")"
]
},
{
"cell_type": "markdown",
"id": "576bd2b4",
"metadata": {
"lines_to_next_cell": 2,
"papermill": {
"duration": 0.001283,
"end_time": "2026-06-13T03:31:21.603128+00:00",
"exception": false,
"start_time": "2026-06-13T03:31:21.601845+00:00",
"status": "completed"
}
},
"source": [
"## 2. Aggregate to Bars\n",
"\n",
"Trade-based features work on aggregated bars (not tick-by-tick).\n",
"Common intervals: 1m, 5m, 15m for intraday; daily for cross-sectional."
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "bdca4846",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T03:31:21.606353Z",
"iopub.status.busy": "2026-06-13T03:31:21.606267Z",
"iopub.status.idle": "2026-06-13T03:31:21.617473Z",
"shell.execute_reply": "2026-06-13T03:31:21.617142Z"
},
"papermill": {
"duration": 0.01361,
"end_time": "2026-06-13T03:31:21.618078+00:00",
"exception": false,
"start_time": "2026-06-13T03:31:21.604468+00:00",
"status": "completed"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Aggregated to 234 bars\n",
"Columns: ['stock', 'timestamp', 'open', 'high', 'low', 'close', 'volume', 'trade_count', 'returns', 'dollar_volume']\n",
"\n",
"AAPL: 77 bars\n"
]
}
],
"source": [
"def aggregate_to_bars(\n",
" trades: pl.DataFrame,\n",
" interval: str = \"5m\",\n",
" stock_col: str = \"stock\",\n",
" price_col: str = \"price\",\n",
" volume_col: str = \"shares\",\n",
") -> pl.DataFrame:\n",
" \"\"\"Aggregate trades to OHLCV bars with volume breakdown.\"\"\"\n",
" bars = (\n",
" trades.sort([stock_col, \"timestamp\"])\n",
" .group_by_dynamic(\"timestamp\", every=interval, by=stock_col)\n",
" .agg(\n",
" [\n",
" pl.col(price_col).first().alias(\"open\"),\n",
" pl.col(price_col).max().alias(\"high\"),\n",
" pl.col(price_col).min().alias(\"low\"),\n",
" pl.col(price_col).last().alias(\"close\"),\n",
" pl.col(volume_col).sum().alias(\"volume\"),\n",
" pl.len().alias(\"trade_count\"),\n",
" ]\n",
" )\n",
" .sort([stock_col, \"timestamp\"])\n",
" )\n",
"\n",
" # Add returns and dollar volume\n",
" return bars.with_columns(\n",
" [\n",
" (pl.col(\"close\") / pl.col(\"close\").shift(1).over(stock_col) - 1).alias(\"returns\"),\n",
" (pl.col(\"close\") * pl.col(\"volume\")).alias(\"dollar_volume\"),\n",
" ]\n",
" )\n",
"\n",
"\n",
"# Create 5-minute bars\n",
"bars = aggregate_to_bars(trades, interval=\"5m\")\n",
"print(f\"Aggregated to {len(bars):,} bars\")\n",
"print(f\"Columns: {bars.columns}\")\n",
"\n",
"# Focus on one stock for visualization\n",
"FOCUS_STOCK = \"AAPL\"\n",
"focus_bars = bars.filter(pl.col(\"stock\") == FOCUS_STOCK).drop_nulls([\"returns\"])\n",
"print(f\"\\n{FOCUS_STOCK}: {len(focus_bars):,} bars\")"
]
},
{
"cell_type": "markdown",
"id": "78569faf",
"metadata": {
"papermill": {
"duration": 0.001972,
"end_time": "2026-06-13T03:31:21.622608+00:00",
"exception": false,
"start_time": "2026-06-13T03:31:21.620636+00:00",
"status": "completed"
}
},
"source": [
"## 3. Trade-Based Liquidity Features\n",
"\n",
"These features require only OHLCV bars (widely available).\n",
"They proxy for market liquidity and trading costs.\n",
"\n",
"| Feature | Formula | Interpretation |\n",
"|---------|---------|----------------|\n",
"| Kyle λ | Cov(ΔP, V) / Var(V) | Price impact per unit volume |\n",
"| Amihud | \\|r\\| / DollarVol | Illiquidity ratio |\n",
"| Roll Spread | 2√(-Cov(ΔP_t, ΔP_{t-1})) | Implied bid-ask spread |"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "af7675b3",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-13T03:31:21.625767Z",
"iopub.status.busy": "2026-06-13T03:31:21.625686Z",
"iopub.status.idle": "2026-06-13T03:31:22.333321Z",
"shell.execute_reply": "2026-06-13T03:31:22.332611Z"
},
"papermill": {
"duration": 0.709757,
"end_time": "2026-06-13T03:31:22.333647+00:00",
"exception": false,
"start_time": "2026-06-13T03:31:21.623890+00:00",
"status": "completed"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Trade-based features computed:\n"
]
},
{
"data": {
"text/html": [
"\n",
"
shape: (10, 6)| timestamp | close | kyle_lambda | amihud | roll_spread | ofi |
|---|
| datetime[ns] | f64 | f64 | f64 | f64 | f64 |
| 2020-01-30 15:10:00 | 321.08 | 0.001004 | 0.000565 | 0.316581 | 1.0 |
| 2020-01-30 15:15:00 | 321.71 | 0.001066 | 0.000565 | 0.235676 | 1.0 |
| 2020-01-30 15:20:00 | 322.23 | 0.001143 | 0.000586 | 0.064782 | 1.0 |
| 2020-01-30 15:25:00 | 321.77 | 0.001174 | 0.000557 | 0.207022 | -1.0 |
| 2020-01-30 15:30:00 | 322.05 | 0.001106 | 0.000494 | 0.214769 | 1.0 |
| 2020-01-30 15:35:00 | 322.355 | 0.00113 | 0.000487 | 0.197683 | 1.0 |
| 2020-01-30 15:40:00 | 322.72 | 0.001154 | 0.000492 | 0.174818 | 1.0 |
| 2020-01-30 15:45:00 | 323.085 | 0.001169 | 0.000484 | 0.156797 | 1.0 |
| 2020-01-30 15:50:00 | 323.38 | 0.001077 | 0.000409 | 0.222852 | 1.0 |
| 2020-01-30 15:55:00 | 323.89 | 0.001039 | 0.000375 | 0.246483 | 1.0 |
\n",
"
shape: (3, 5)| stock | kyle_median | amihud_median | total_volume | n_bars |
|---|
| str | f64 | f64 | u32 | u32 |
| "AAPL" | 0.001167 | 0.000561 | 414061 | 40 |
| "MSFT" | 0.001604 | 0.000415 | 1140077 | 58 |
| "TSLA" | 0.003053 | 0.000172 | 1144961 | 58 |