wehub-resource-sync
1120 lines
40 KiB
Plaintext
1120 lines
40 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "e56eade9",
|
||
"metadata": {
|
||
"tags": []
|
||
},
|
||
"source": [
|
||
"# NASDAQ TotalView-ITCH: Order Book Data Parsing\n",
|
||
"\n",
|
||
"**Chapter 3: Market Microstructure**\n",
|
||
"\n",
|
||
"**Docker image**: `ml4t`\n",
|
||
"\n",
|
||
"## Purpose\n",
|
||
"\n",
|
||
"This notebook demonstrates how to parse NASDAQ's TotalView-ITCH binary protocol.\n",
|
||
"Understanding MBO (message-by-order) data is foundational for microstructure-based ML features.\n",
|
||
"\n",
|
||
"## Learning Objectives\n",
|
||
"\n",
|
||
"After completing this notebook, you will be able to:\n",
|
||
"- Understand the ITCH 5.0 binary message format\n",
|
||
"- Parse ITCH messages using Python's `struct` module\n",
|
||
"- Load pre-parsed ITCH data for analysis\n",
|
||
"- Choose between Python (educational) and Rust (production) parsers\n",
|
||
"\n",
|
||
"## Cross-References\n",
|
||
"\n",
|
||
"- **Downstream**: `02_itch_lob_reconstruction` (builds order book from these messages)\n",
|
||
"- **Related**: `09_databento_mbo_analysis` (alternative MBO data source)\n",
|
||
"\n",
|
||
"## Data Requirements\n",
|
||
"\n",
|
||
"ITCH sample data can be downloaded using:\n",
|
||
"```bash\n",
|
||
"python data/equities/market/microstructure/nasdaq_itch_download.py --list # List available files\n",
|
||
"python data/equities/market/microstructure/nasdaq_itch_download.py # Download default sample\n",
|
||
"```\n",
|
||
"\n",
|
||
"Files are ~5GB compressed from: https://emi.nasdaq.com/ITCH/"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "433ce285",
|
||
"metadata": {
|
||
"tags": []
|
||
},
|
||
"source": [
|
||
"## ITCH Message Types\n",
|
||
"\n",
|
||
"The [ITCH v5.0 specification](https://www.nasdaqtrader.com/content/technicalsupport/specifications/dataproducts/NQTVITCHSpecification.pdf) defines 20+ message types:\n",
|
||
"\n",
|
||
"| Type | Name | Description |\n",
|
||
"|------|------|-------------|\n",
|
||
"| **S** | System Event | Market open/close events |\n",
|
||
"| **R** | Stock Directory | Ticker information and characteristics |\n",
|
||
"| **H** | Trading Action | Trading halts, pauses, and resumptions |\n",
|
||
"| **Y** | Reg SHO Restriction | Short sale price test restrictions |\n",
|
||
"| **L** | Market Participant | Market maker positions |\n",
|
||
"| **V** | MWCB Decline Level | Market-wide circuit breaker levels |\n",
|
||
"| **W** | MWCB Status | Circuit breaker breach status |\n",
|
||
"| **A** | Add Order | New limit order enters the book |\n",
|
||
"| **F** | Add Order (MPID) | Same as A, with market participant ID |\n",
|
||
"| **E** | Order Executed | Partial/full execution against standing order |\n",
|
||
"| **C** | Order Executed w/Price | Execution at different price (hidden orders) |\n",
|
||
"| **X** | Order Cancel | Partial cancellation |\n",
|
||
"| **D** | Order Delete | Full removal from book |\n",
|
||
"| **U** | Order Replace | Modify price/size (cancel + add) |\n",
|
||
"| **P** | Trade | Non-displayed execution |\n",
|
||
"| **Q** | Cross Trade | Opening/closing cross |\n",
|
||
"| **B** | Broken Trade | Trade cancellation |\n",
|
||
"| **I** | NOII | Net Order Imbalance Indicator (auction) |\n",
|
||
"| **J** | LULD Auction Collar | Limit up-limit down price bands |\n",
|
||
"| **K** | IPO Quoting Period | IPO quotation timing |\n",
|
||
"\n",
|
||
"By combining these messages chronologically, we can reconstruct the order book at any point in time."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 1,
|
||
"id": "1703fb09",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-05-15T19:16:21.269661Z",
|
||
"iopub.status.busy": "2026-05-15T19:16:21.269440Z",
|
||
"iopub.status.idle": "2026-05-15T19:16:22.641043Z",
|
||
"shell.execute_reply": "2026-05-15T19:16:22.640726Z"
|
||
},
|
||
"tags": []
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"\"\"\"NASDAQ TotalView-ITCH: Order Book Data Parsing — parse ITCH binary protocol into structured messages.\"\"\"\n",
|
||
"\n",
|
||
"import gzip\n",
|
||
"import os\n",
|
||
"import shutil\n",
|
||
"import struct\n",
|
||
"import warnings\n",
|
||
"from collections import Counter, defaultdict\n",
|
||
"from datetime import date, datetime\n",
|
||
"from pathlib import Path\n",
|
||
"from time import time\n",
|
||
"\n",
|
||
"warnings.filterwarnings(\"ignore\")\n",
|
||
"\n",
|
||
"import polars as pl\n",
|
||
"from itch_message_specs import (\n",
|
||
" FMT_DICT,\n",
|
||
" MESSAGE_SPECS,\n",
|
||
" NT_DICT,\n",
|
||
" flush_to_parquet,\n",
|
||
" parse_price4,\n",
|
||
" parse_timestamp,\n",
|
||
" print_message_formats,\n",
|
||
")\n",
|
||
"from tqdm.auto import tqdm\n",
|
||
"\n",
|
||
"from data import load_nasdaq_itch"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 2,
|
||
"id": "e81ca2a8",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-05-15T19:16:22.645110Z",
|
||
"iopub.status.busy": "2026-05-15T19:16:22.645020Z",
|
||
"iopub.status.idle": "2026-05-15T19:16:22.646574Z",
|
||
"shell.execute_reply": "2026-05-15T19:16:22.646343Z"
|
||
},
|
||
"tags": [
|
||
"parameters"
|
||
]
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"SKIP_PARSING = False"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 4,
|
||
"id": "c37cb809",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-05-15T19:16:22.655152Z",
|
||
"iopub.status.busy": "2026-05-15T19:16:22.655077Z",
|
||
"iopub.status.idle": "2026-05-15T19:16:22.657469Z",
|
||
"shell.execute_reply": "2026-05-15T19:16:22.657228Z"
|
||
},
|
||
"tags": []
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Raw ITCH data (input): data/equities/market/microstructure/nasdaq_itch/raw\n",
|
||
"Parsed messages (output): data/equities/market/microstructure/nasdaq_itch/messages\n",
|
||
"Raw data exists: True\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Data paths\n",
|
||
"# Canonical parsed messages location (both read and write target)\n",
|
||
"MESSAGE_DIR = load_nasdaq_itch(get_base_path=True)\n",
|
||
"MESSAGE_DIR.mkdir(parents=True, exist_ok=True)\n",
|
||
"\n",
|
||
"# Raw ITCH binary input (from download script)\n",
|
||
"ITCH_RAW_DIR = MESSAGE_DIR.parent / \"raw\"\n",
|
||
"\n",
|
||
"print(f\"Raw ITCH data (input): {ITCH_RAW_DIR}\")\n",
|
||
"print(f\"Parsed messages (output): {MESSAGE_DIR}\")\n",
|
||
"_raw_present = ITCH_RAW_DIR.exists() and next(ITCH_RAW_DIR.iterdir(), None) is not None\n",
|
||
"print(f\"Raw data exists: {_raw_present}\")\n",
|
||
"if not _raw_present:\n",
|
||
" print(\" (run NB00_itch_download.py first to fetch the ITCH binary)\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "bd4a150c",
|
||
"metadata": {
|
||
"tags": []
|
||
},
|
||
"source": [
|
||
"## 1. Message Specifications\n",
|
||
"\n",
|
||
"Each ITCH message has a fixed binary structure. The format is defined in `itch_message_specs.py`\n",
|
||
"using Python's `struct` module. Format codes:\n",
|
||
"- `H` = unsigned short (2 bytes)\n",
|
||
"- `I` = unsigned int (4 bytes)\n",
|
||
"- `Q` = unsigned long long (8 bytes)\n",
|
||
"- `s` = char (1 byte), `Ns` = N chars\n",
|
||
"- `>` = big-endian byte order"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 5,
|
||
"id": "72896c08",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-05-15T19:16:22.662724Z",
|
||
"iopub.status.busy": "2026-05-15T19:16:22.662658Z",
|
||
"iopub.status.idle": "2026-05-15T19:16:22.664358Z",
|
||
"shell.execute_reply": "2026-05-15T19:16:22.664017Z"
|
||
},
|
||
"tags": []
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"ITCH Message Formats:\n",
|
||
"\n",
|
||
" S (System Event ): 11 bytes - >HH6ss\n",
|
||
" R (Stock Directory ): 38 bytes - >HH6s8sssIss2ssssssIs\n",
|
||
" A (Add Order ): 35 bytes - >HH6sQsI8sI\n",
|
||
" F (Add Order MPID ): 39 bytes - >HH6sQsI8sI4s\n",
|
||
" E (Order Executed ): 30 bytes - >HH6sQIQ\n",
|
||
" C (Order Executed with Price): 35 bytes - >HH6sQIQsI\n",
|
||
" X (Order Cancel ): 22 bytes - >HH6sQI\n",
|
||
" D (Order Delete ): 18 bytes - >HH6sQ\n",
|
||
" U (Order Replace ): 34 bytes - >HH6sQQII\n",
|
||
" P (Trade ): 43 bytes - >HH6sQsI8sIQ\n",
|
||
" Q (Cross Trade ): 39 bytes - >HH6sQ8sIQs\n",
|
||
" H (Trading Action ): 24 bytes - >HH6s8sss4s\n",
|
||
" Y (Reg SHO Restriction ): 19 bytes - >HH6s8ss\n",
|
||
" L (Market Participant Position): 25 bytes - >HH6s4s8ssss\n",
|
||
" V (MWCB Decline Level ): 34 bytes - >HH6sQQQ\n",
|
||
" W (MWCB Status ): 11 bytes - >HH6ss\n",
|
||
" J (LULD Auction Collar ): 34 bytes - >HH6s8sIIII\n",
|
||
" K (IPO Quoting Period ): 27 bytes - >HH6s8sIsI\n",
|
||
" B (Broken Trade ): 18 bytes - >HH6sQ\n",
|
||
" I (NOII ): 49 bytes - >HH6sQQs8sIIIss\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Show message formats (loaded from utils/itch_message_specs.py)\n",
|
||
"print_message_formats()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "df5bfd12",
|
||
"metadata": {
|
||
"tags": []
|
||
},
|
||
"source": [
|
||
"## 2. Binary Parsing Example\n",
|
||
"\n",
|
||
"Let's demonstrate how binary parsing works by creating and parsing a sample Add Order message."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 6,
|
||
"id": "4d7398e4",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-05-15T19:16:22.670553Z",
|
||
"iopub.status.busy": "2026-05-15T19:16:22.670441Z",
|
||
"iopub.status.idle": "2026-05-15T19:16:22.673056Z",
|
||
"shell.execute_reply": "2026-05-15T19:16:22.672624Z"
|
||
},
|
||
"tags": []
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Raw Add Order message (35 bytes):\n",
|
||
" Hex: 04d2162e000000000001000000024cb016ea42000000644141504c202020200016e360\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Create a sample Add Order message to demonstrate parsing\n",
|
||
"# Format: >HH6sQsI8sI (big-endian)\n",
|
||
"\n",
|
||
"sample_add_order = struct.pack(\n",
|
||
" \">HH6sQsI8sI\",\n",
|
||
" 1234, # stock_locate\n",
|
||
" 5678, # tracking_number\n",
|
||
" b\"\\x00\\x00\\x00\\x00\\x00\\x01\", # timestamp (1 nanosecond)\n",
|
||
" 9876543210, # order_reference_number\n",
|
||
" b\"B\", # buy_sell_indicator\n",
|
||
" 100, # shares\n",
|
||
" b\"AAPL \", # stock (padded to 8 chars)\n",
|
||
" 1500000, # price (150.0000 in price4 format)\n",
|
||
")\n",
|
||
"\n",
|
||
"print(f\"Raw Add Order message ({len(sample_add_order)} bytes):\")\n",
|
||
"print(f\" Hex: {sample_add_order.hex()}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 7,
|
||
"id": "5dd38245",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-05-15T19:16:22.676322Z",
|
||
"iopub.status.busy": "2026-05-15T19:16:22.676239Z",
|
||
"iopub.status.idle": "2026-05-15T19:16:22.678716Z",
|
||
"shell.execute_reply": "2026-05-15T19:16:22.678318Z"
|
||
},
|
||
"tags": []
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Parsed Add Order Message:\n",
|
||
"----------------------------------------\n",
|
||
" stock_locate : 1234\n",
|
||
" tracking_number : 5678\n",
|
||
" timestamp : \u0000\u0000\u0000\u0000\u0000\u0001\n",
|
||
" order_reference_number : 9876543210\n",
|
||
" buy_sell_indicator : B\n",
|
||
" shares : 100\n",
|
||
" stock : AAPL\n",
|
||
" price : 1500000\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Parse the binary data using our struct format\n",
|
||
"parsed = struct.unpack(FMT_DICT[\"A\"], sample_add_order)\n",
|
||
"add_order = NT_DICT[\"A\"]._make(parsed)\n",
|
||
"\n",
|
||
"print(\"Parsed Add Order Message:\")\n",
|
||
"print(\"-\" * 40)\n",
|
||
"for field, value in add_order._asdict().items():\n",
|
||
" if isinstance(value, bytes):\n",
|
||
" value = value.decode(\"ascii\").strip()\n",
|
||
" print(f\" {field:25}: {value}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 8,
|
||
"id": "e386307b",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-05-15T19:16:22.682144Z",
|
||
"iopub.status.busy": "2026-05-15T19:16:22.682039Z",
|
||
"iopub.status.idle": "2026-05-15T19:16:22.685672Z",
|
||
"shell.execute_reply": "2026-05-15T19:16:22.684588Z"
|
||
},
|
||
"tags": []
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Timestamp: 1 nanoseconds = 0.000000001 seconds after midnight\n",
|
||
"Price: $150.0000\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Apply conversions using helper functions from utils.itch_message_specs\n",
|
||
"ts_ns = parse_timestamp(add_order.timestamp)\n",
|
||
"price = parse_price4(add_order.price)\n",
|
||
"\n",
|
||
"print(f\"Timestamp: {ts_ns:,} nanoseconds = {ts_ns / 1e9:.9f} seconds after midnight\")\n",
|
||
"print(f\"Price: ${price:.4f}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "360596c3",
|
||
"metadata": {
|
||
"tags": []
|
||
},
|
||
"source": [
|
||
"## 3. Loading Pre-Parsed ITCH Data\n",
|
||
"\n",
|
||
"If you've already parsed ITCH data (using the Rust parser or Python parser), you can load\n",
|
||
"the pre-parsed messages directly. This is the recommended approach for analysis."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 9,
|
||
"id": "1f9b3c60",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-05-15T19:16:22.693467Z",
|
||
"iopub.status.busy": "2026-05-15T19:16:22.693361Z",
|
||
"iopub.status.idle": "2026-05-15T19:16:22.710132Z",
|
||
"shell.execute_reply": "2026-05-15T19:16:22.709729Z"
|
||
},
|
||
"tags": []
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"ITCH Data Pipeline Status:\n",
|
||
"--------------------------------------------------\n",
|
||
"Raw binary files: 2\n",
|
||
" 01302020.NASDAQ_ITCH50.gz (5.60 GB)\n",
|
||
" 01302020.NASDAQ_ITCH50.bin (12.95 GB)\n",
|
||
"Parsed message types: 18\n",
|
||
" A (Add Order): 43 files\n",
|
||
" C (Order Executed with Price): 43 files\n",
|
||
" D (Order Delete): 43 files\n",
|
||
" E (Order Executed): 43 files\n",
|
||
" F (Add Order MPID): 43 files\n",
|
||
" H (Trading Action): 11 files\n",
|
||
" I (NOII): 18 files\n",
|
||
" J (LULD Auction Collar): 4 files\n",
|
||
" K (IPO Quoting Period): 2 files\n",
|
||
" L (Market Participant Position): 21 files\n",
|
||
" P (Trade): 43 files\n",
|
||
" Q (Cross Trade): 7 files\n",
|
||
" R (Stock Directory): 1 files\n",
|
||
" S (System Event): 4 files\n",
|
||
" U (Order Replace): 43 files\n",
|
||
" V (MWCB Decline Level): 1 files\n",
|
||
" X (Order Cancel): 43 files\n",
|
||
" Y (Reg SHO Restriction): 38 files\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Check what data is available locally\n",
|
||
"print(\"ITCH Data Pipeline Status:\")\n",
|
||
"print(\"-\" * 50)\n",
|
||
"\n",
|
||
"# Step 1: Raw binary from download\n",
|
||
"raw_files = []\n",
|
||
"if ITCH_RAW_DIR.exists():\n",
|
||
" raw_files = list(ITCH_RAW_DIR.glob(\"*.gz\")) + list(ITCH_RAW_DIR.glob(\"*.bin\"))\n",
|
||
"print(f\"Raw binary files: {len(raw_files)}\")\n",
|
||
"for f in raw_files:\n",
|
||
" print(f\" {f.name} ({f.stat().st_size / 1e9:.2f} GB)\")\n",
|
||
"\n",
|
||
"# Step 2: Parsed messages (single uppercase letter = message type)\n",
|
||
"parsed_types = (\n",
|
||
" [\n",
|
||
" d\n",
|
||
" for d in sorted(MESSAGE_DIR.iterdir())\n",
|
||
" if d.is_dir() and len(d.name) == 1 and d.name.isupper()\n",
|
||
" ]\n",
|
||
" if MESSAGE_DIR.exists()\n",
|
||
" else []\n",
|
||
")\n",
|
||
"parsed_with_data = [d for d in parsed_types if list(d.glob(\"*.parquet\"))]\n",
|
||
"print(f\"Parsed message types: {len(parsed_with_data)}\")\n",
|
||
"for msg_dir in parsed_with_data:\n",
|
||
" name = MESSAGE_SPECS.get(msg_dir.name, {}).get(\"name\", \"Unknown\")\n",
|
||
" n_files = len(list(msg_dir.glob(\"*.parquet\")))\n",
|
||
" print(f\" {msg_dir.name} ({name}): {n_files} files\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 10,
|
||
"id": "aaa34833",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-05-15T19:16:22.713851Z",
|
||
"iopub.status.busy": "2026-05-15T19:16:22.713765Z",
|
||
"iopub.status.idle": "2026-05-15T19:16:22.748084Z",
|
||
"shell.execute_reply": "2026-05-15T19:16:22.747543Z"
|
||
},
|
||
"tags": []
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"\n",
|
||
"Loaded 1,779,727 trade messages\n",
|
||
"Columns: ['stock_locate', 'tracking_number', 'timestamp', 'order_reference_number', 'buy_sell_indicator', 'shares', 'stock', 'price', 'match_number']\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Validate: at minimum we need parsed data to continue\n",
|
||
"trade_dir = MESSAGE_DIR / \"P\"\n",
|
||
"assert trade_dir.exists() and list(trade_dir.glob(\"*.parquet\")), (\n",
|
||
" f\"No parsed ITCH data at {MESSAGE_DIR}.\\n\"\n",
|
||
" \"To set up the data pipeline:\\n\"\n",
|
||
" \" 1. Download raw data: uv run python data/equities/market/microstructure/nasdaq_itch_download.py\\n\"\n",
|
||
" \" 2. Parse (this notebook, Section 4) or use Rust parser (Section 6)\\n\"\n",
|
||
" \" 3. Parsed messages go to: data/equities/market/microstructure/nasdaq_itch/messages/\"\n",
|
||
")\n",
|
||
"\n",
|
||
"trades = pl.read_parquet(trade_dir / \"*.parquet\")\n",
|
||
"print(f\"\\nLoaded {len(trades):,} trade messages\")\n",
|
||
"print(f\"Columns: {trades.columns}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "7adca783",
|
||
"metadata": {
|
||
"lines_to_next_cell": 2,
|
||
"tags": []
|
||
},
|
||
"source": [
|
||
"## 4. Full Parser Implementation\n",
|
||
"\n",
|
||
"This Python parser is for **educational purposes**. For production use with large files,\n",
|
||
"use the Rust parser (see Section 6) which provides order-of-magnitude speedups."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "f1879b4c",
|
||
"metadata": {
|
||
"lines_to_next_cell": 2,
|
||
"tags": []
|
||
},
|
||
"source": [
|
||
"### Parser Helpers\n",
|
||
"\n",
|
||
"We split the parser into three functions: `_read_frame` reads one binary message\n",
|
||
"frame, `_decode_message` unpacks and converts it, and `parse_itch_file` orchestrates\n",
|
||
"the loop with buffered Parquet writes."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 11,
|
||
"id": "fc171eb9",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-05-15T19:16:22.758455Z",
|
||
"iopub.status.busy": "2026-05-15T19:16:22.758345Z",
|
||
"iopub.status.idle": "2026-05-15T19:16:22.760779Z",
|
||
"shell.execute_reply": "2026-05-15T19:16:22.760500Z"
|
||
},
|
||
"lines_to_next_cell": 2,
|
||
"tags": []
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"def _read_frame(f, pbar) -> tuple[str, bytes] | None:\n",
|
||
" \"\"\"Read one ITCH message frame: 2-byte length + 1-byte type + payload.\n",
|
||
"\n",
|
||
" Returns (msg_type, payload) on success, or None on EOF/truncation.\n",
|
||
" \"\"\"\n",
|
||
" # 2-byte big-endian length prefix (message size including type byte)\n",
|
||
" length_bytes = f.read(2)\n",
|
||
" if len(length_bytes) < 2:\n",
|
||
" return None\n",
|
||
" pbar.update(2)\n",
|
||
"\n",
|
||
" msg_size = int.from_bytes(length_bytes, \"big\")\n",
|
||
"\n",
|
||
" # 1-byte message type\n",
|
||
" msg_type_byte = f.read(1)\n",
|
||
" if len(msg_type_byte) < 1:\n",
|
||
" print(f\"\\nWarning: Truncated message at byte {f.tell()}, expected type byte\")\n",
|
||
" return None\n",
|
||
" pbar.update(1)\n",
|
||
"\n",
|
||
" msg_type = msg_type_byte.decode(\"ascii\")\n",
|
||
"\n",
|
||
" # Payload (msg_size includes type byte, so payload is msg_size - 1)\n",
|
||
" payload = f.read(msg_size - 1)\n",
|
||
" if len(payload) < msg_size - 1:\n",
|
||
" print(f\"\\nWarning: Truncated payload for message type {msg_type}\")\n",
|
||
" return None\n",
|
||
" pbar.update(msg_size - 1)\n",
|
||
"\n",
|
||
" return msg_type, payload"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "fa0c6af1",
|
||
"metadata": {
|
||
"lines_to_next_cell": 2,
|
||
"tags": []
|
||
},
|
||
"source": [
|
||
"Decode binary payload into a Python dict, converting raw timestamp bytes\n",
|
||
"to nanosecond integers and byte strings to stripped ASCII."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 12,
|
||
"id": "04095c69",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-05-15T19:16:22.766968Z",
|
||
"iopub.status.busy": "2026-05-15T19:16:22.766820Z",
|
||
"iopub.status.idle": "2026-05-15T19:16:22.769347Z",
|
||
"shell.execute_reply": "2026-05-15T19:16:22.769072Z"
|
||
},
|
||
"lines_to_next_cell": 2,
|
||
"tags": []
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"def _decode_message(msg_type: str, payload: bytes) -> dict | None:\n",
|
||
" \"\"\"Unpack binary payload into a dict, converting timestamps and strings.\n",
|
||
"\n",
|
||
" Returns parsed message dict, or None on struct error.\n",
|
||
" \"\"\"\n",
|
||
" try:\n",
|
||
" parsed = struct.unpack(FMT_DICT[msg_type], payload)\n",
|
||
" msg = NT_DICT[msg_type]._make(parsed)._asdict()\n",
|
||
" except struct.error:\n",
|
||
" return None\n",
|
||
"\n",
|
||
" # Convert timestamp: nanoseconds since midnight\n",
|
||
" if \"timestamp\" in msg:\n",
|
||
" msg[\"timestamp\"] = int.from_bytes(msg[\"timestamp\"], \"big\")\n",
|
||
"\n",
|
||
" # Decode string fields\n",
|
||
" for field, value in msg.items():\n",
|
||
" if isinstance(value, bytes):\n",
|
||
" msg[field] = value.decode(\"ascii\").strip()\n",
|
||
"\n",
|
||
" return msg"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "48265d95",
|
||
"metadata": {
|
||
"lines_to_next_cell": 2,
|
||
"tags": []
|
||
},
|
||
"source": [
|
||
"The main parser reads the binary file sequentially, buffering decoded messages\n",
|
||
"and flushing to Parquet periodically to bound memory usage."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 13,
|
||
"id": "56e2f489",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-05-15T19:16:22.775469Z",
|
||
"iopub.status.busy": "2026-05-15T19:16:22.775382Z",
|
||
"iopub.status.idle": "2026-05-15T19:16:22.778878Z",
|
||
"shell.execute_reply": "2026-05-15T19:16:22.778583Z"
|
||
},
|
||
"tags": [],
|
||
"title": "— single function body, helpers already extracted"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"def parse_itch_file(\n",
|
||
" itch_file: Path,\n",
|
||
" trading_day: date,\n",
|
||
" output_dir: Path,\n",
|
||
" max_buffered_messages: int = 10_000_000,\n",
|
||
" max_messages: int | None = None,\n",
|
||
") -> dict[str, int]:\n",
|
||
" \"\"\"Parse ITCH binary file and store messages as Parquet.\n",
|
||
"\n",
|
||
" Args:\n",
|
||
" itch_file: Path to binary ITCH file (.bin, not .gz).\n",
|
||
" trading_day: Trading date for timestamp construction.\n",
|
||
" output_dir: Directory for Parquet output (one subdir per message type).\n",
|
||
" max_buffered_messages: Flush threshold (total buffered messages).\n",
|
||
" max_messages: Optional limit for testing.\n",
|
||
"\n",
|
||
" Returns:\n",
|
||
" Dictionary with message type counts.\n",
|
||
" \"\"\"\n",
|
||
" # Midnight timestamp for the trading day (ITCH timestamps are nanoseconds offset)\n",
|
||
" base_ts = datetime(trading_day.year, trading_day.month, trading_day.day)\n",
|
||
" file_counters: dict[str, int] = defaultdict(int)\n",
|
||
"\n",
|
||
" buffers = defaultdict(list)\n",
|
||
" counts = Counter()\n",
|
||
" file_size = itch_file.stat().st_size\n",
|
||
" start_time = time()\n",
|
||
"\n",
|
||
" with (\n",
|
||
" itch_file.open(\"rb\") as f,\n",
|
||
" tqdm(total=file_size, desc=\"Parsing ITCH\", unit=\"B\", unit_scale=True) as pbar,\n",
|
||
" ):\n",
|
||
" while True:\n",
|
||
" if max_messages and sum(counts.values()) >= max_messages:\n",
|
||
" print(f\"\\nLimit reached: {max_messages:,} messages\")\n",
|
||
" break\n",
|
||
"\n",
|
||
" frame = _read_frame(f, pbar)\n",
|
||
" if frame is None:\n",
|
||
" break\n",
|
||
" msg_type, payload = frame\n",
|
||
" counts[msg_type] += 1\n",
|
||
"\n",
|
||
" if msg_type not in FMT_DICT:\n",
|
||
" continue\n",
|
||
"\n",
|
||
" msg = _decode_message(msg_type, payload)\n",
|
||
" if msg is None:\n",
|
||
" continue\n",
|
||
"\n",
|
||
" # Check for end of messages\n",
|
||
" if msg_type == \"S\" and msg.get(\"event_code\") == \"C\":\n",
|
||
" print(\"\\nEnd of Messages\")\n",
|
||
" flush_to_parquet(buffers, output_dir, base_ts, file_counters)\n",
|
||
" break\n",
|
||
"\n",
|
||
" buffers[msg_type].append(msg)\n",
|
||
"\n",
|
||
" # Periodic flush\n",
|
||
" if sum(len(v) for v in buffers.values()) >= max_buffered_messages:\n",
|
||
" flush_to_parquet(buffers, output_dir, base_ts, file_counters)\n",
|
||
"\n",
|
||
" # Final flush\n",
|
||
" if any(buffers.values()):\n",
|
||
" flush_to_parquet(buffers, output_dir, base_ts, file_counters)\n",
|
||
"\n",
|
||
" elapsed = time() - start_time\n",
|
||
" total = sum(counts.values())\n",
|
||
" print(f\"Parsed {total:,} messages in {elapsed:.1f}s ({total / elapsed:,.0f} msg/s)\")\n",
|
||
"\n",
|
||
" return dict(counts)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 14,
|
||
"id": "31501342",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-05-15T19:16:22.782343Z",
|
||
"iopub.status.busy": "2026-05-15T19:16:22.782220Z",
|
||
"iopub.status.idle": "2026-05-15T19:16:22.785693Z",
|
||
"shell.execute_reply": "2026-05-15T19:16:22.785325Z"
|
||
},
|
||
"tags": []
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"SKIP_PARSING=True: skipping ITCH binary parsing (uses pre-parsed data)\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Locate ITCH data file (skip if SKIP_PARSING is set)\n",
|
||
"if SKIP_PARSING:\n",
|
||
" print(\"SKIP_PARSING=True: skipping ITCH binary parsing (uses pre-parsed data)\")\n",
|
||
" itch_file = None\n",
|
||
" counts = {}\n",
|
||
"else:\n",
|
||
" # Clear any existing parsed data to avoid schema conflicts\n",
|
||
" # (Different parser versions may produce different schemas)\n",
|
||
" # Set ITCH_KEEP_EXISTING=1 to skip cleanup and use existing data\n",
|
||
" clear_existing = os.environ.get(\"ITCH_KEEP_EXISTING\", \"0\") != \"1\"\n",
|
||
" if clear_existing and MESSAGE_DIR.exists() and list(MESSAGE_DIR.glob(\"*/part-*.parquet\")):\n",
|
||
" print(f\"Clearing existing parsed data in {MESSAGE_DIR}\")\n",
|
||
" print(\" (Set ITCH_KEEP_EXISTING=1 to keep existing data)\")\n",
|
||
" shutil.rmtree(MESSAGE_DIR)\n",
|
||
" MESSAGE_DIR.mkdir(parents=True, exist_ok=True)\n",
|
||
"\n",
|
||
" # Find ITCH file (compressed or uncompressed)\n",
|
||
" gz_files = list(ITCH_RAW_DIR.glob(\"*.gz\")) if ITCH_RAW_DIR.exists() else []\n",
|
||
" bin_files = list(ITCH_RAW_DIR.glob(\"*.bin\")) if ITCH_RAW_DIR.exists() else []\n",
|
||
"\n",
|
||
" if not gz_files and not bin_files:\n",
|
||
" raise FileNotFoundError(\n",
|
||
" f\"No raw ITCH binary found at {ITCH_RAW_DIR}.\\n\"\n",
|
||
" \"Download first:\\n\"\n",
|
||
" \" uv run python data/equities/market/microstructure/nasdaq_itch_download.py\"\n",
|
||
" )"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 15,
|
||
"id": "1049de99",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-05-15T19:16:22.789206Z",
|
||
"iopub.status.busy": "2026-05-15T19:16:22.789135Z",
|
||
"iopub.status.idle": "2026-05-15T19:16:22.792071Z",
|
||
"shell.execute_reply": "2026-05-15T19:16:22.791787Z"
|
||
},
|
||
"tags": []
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Decompress if needed and extract trading date\n",
|
||
"if not SKIP_PARSING and (gz_files or bin_files):\n",
|
||
" # Prefer uncompressed, otherwise decompress\n",
|
||
" if bin_files:\n",
|
||
" itch_file = bin_files[0]\n",
|
||
" print(f\"Found uncompressed: {itch_file.name}\")\n",
|
||
" else:\n",
|
||
" gz_file = gz_files[0]\n",
|
||
" itch_file = gz_file.with_suffix(\".bin\")\n",
|
||
"\n",
|
||
" if not itch_file.exists():\n",
|
||
" print(f\"Decompressing {gz_file.name}...\")\n",
|
||
" with gzip.open(gz_file, \"rb\") as f_in, open(itch_file, \"wb\") as f_out:\n",
|
||
" shutil.copyfileobj(f_in, f_out)\n",
|
||
" print(f\"Created: {itch_file.name} ({itch_file.stat().st_size / 1e9:.1f} GB)\")\n",
|
||
" else:\n",
|
||
" print(f\"Found: {itch_file.name}\")\n",
|
||
"\n",
|
||
" # Extract trading date from filename (format: MMDDYYYY.NASDAQ_ITCH50.bin)\n",
|
||
" date_str = itch_file.stem.split(\".\")[0]\n",
|
||
" if len(date_str) == 8 and date_str.isdigit():\n",
|
||
" trading_day = date(int(date_str[4:8]), int(date_str[:2]), int(date_str[2:4]))\n",
|
||
" else:\n",
|
||
" trading_day = date(2020, 1, 30) # Fallback\n",
|
||
"\n",
|
||
" print(f\"Trading day: {trading_day}\")\n",
|
||
"\n",
|
||
" # Parse ITCH file (full-day parse: ~22 min on the reference machine)\n",
|
||
" if itch_file and itch_file.exists():\n",
|
||
" counts = parse_itch_file(\n",
|
||
" itch_file=itch_file,\n",
|
||
" trading_day=trading_day,\n",
|
||
" output_dir=MESSAGE_DIR,\n",
|
||
" # max_messages=1_000_000, # Remove this line for full parse\n",
|
||
" )\n",
|
||
"\n",
|
||
" print(\"\\nMessage counts:\")\n",
|
||
" for msg_type, count in sorted(counts.items(), key=lambda x: -x[1]):\n",
|
||
" name = MESSAGE_SPECS.get(msg_type, {}).get(\"name\", \"Unknown\")\n",
|
||
" print(f\" {msg_type} ({name:25}): {count:>10,}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "552ce57f",
|
||
"metadata": {
|
||
"tags": []
|
||
},
|
||
"source": [
|
||
"## 5. Message Type Analysis\n",
|
||
"\n",
|
||
"After parsing, we can analyze message distributions."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 16,
|
||
"id": "2ff9d6e5",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-05-15T19:16:22.798717Z",
|
||
"iopub.status.busy": "2026-05-15T19:16:22.798593Z",
|
||
"iopub.status.idle": "2026-05-15T19:16:22.828000Z",
|
||
"shell.execute_reply": "2026-05-15T19:16:22.827633Z"
|
||
},
|
||
"tags": []
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Parsed Message Types:\n",
|
||
"--------------------------------------------------\n",
|
||
" A (Add Order ): 184,735,355 messages\n",
|
||
" C (Order Executed with Price): 139,474 messages\n",
|
||
" D (Order Delete ): 180,285,101 messages\n",
|
||
" E (Order Executed ): 8,415,610 messages\n",
|
||
" F (Add Order MPID ): 1,875,350 messages\n",
|
||
" H (Trading Action ): 8,921 messages\n",
|
||
" I (NOII ): 4,025,192 messages\n",
|
||
" J (LULD Auction Collar ): 5 messages\n",
|
||
" K (IPO Quoting Period ): 2 messages\n",
|
||
" L (Market Participant Position): 216,802 messages\n",
|
||
" P (Trade ): 1,779,727 messages\n",
|
||
" Q (Cross Trade ): 17,835 messages\n",
|
||
" R (Stock Directory ): 8,916 messages\n",
|
||
" S (System Event ): 5 messages\n",
|
||
" U (Order Replace ): 36,777,372 messages\n",
|
||
" V (MWCB Decline Level ): 1 messages\n",
|
||
" X (Order Cancel ): 4,990,972 messages\n",
|
||
" Y (Reg SHO Restriction ): 9,068 messages\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Message type distribution — use lazy scan to count without loading all data\n",
|
||
"print(\"Parsed Message Types:\")\n",
|
||
"print(\"-\" * 50)\n",
|
||
"for msg_dir in sorted(MESSAGE_DIR.iterdir()):\n",
|
||
" if (\n",
|
||
" msg_dir.is_dir()\n",
|
||
" and len(msg_dir.name) == 1\n",
|
||
" and msg_dir.name.isupper()\n",
|
||
" and list(msg_dir.glob(\"*.parquet\"))\n",
|
||
" ):\n",
|
||
" count = pl.scan_parquet(msg_dir / \"*.parquet\").select(pl.len()).collect().item()\n",
|
||
" name = MESSAGE_SPECS.get(msg_dir.name, {}).get(\"name\", \"Unknown\")\n",
|
||
" print(f\" {msg_dir.name} ({name:25}): {count:>12,} messages\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 17,
|
||
"id": "03398b75",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-05-15T19:16:22.832003Z",
|
||
"iopub.status.busy": "2026-05-15T19:16:22.831896Z",
|
||
"iopub.status.idle": "2026-05-15T19:16:22.871465Z",
|
||
"shell.execute_reply": "2026-05-15T19:16:22.871149Z"
|
||
},
|
||
"tags": []
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Schema Compatibility Check:\n",
|
||
"--------------------------------------------------\n",
|
||
" [OK] A (Add Order ): cols=['stock_locate', 'tracking_number', 'timestamp', 'order_reference_number']...\n",
|
||
" [OK] C (Order Executed with Price): cols=['stock_locate', 'tracking_number', 'timestamp', 'order_reference_number']...\n",
|
||
" [OK] D (Order Delete ): cols=['stock_locate', 'tracking_number', 'timestamp', 'order_reference_number']...\n",
|
||
" [OK] E (Order Executed ): cols=['stock_locate', 'tracking_number', 'timestamp', 'order_reference_number']...\n",
|
||
" [OK] F (Add Order MPID ): cols=['stock_locate', 'tracking_number', 'timestamp', 'order_reference_number']...\n",
|
||
" [OK] H (Trading Action ): cols=['stock_locate', 'tracking_number', 'timestamp', 'stock']...\n",
|
||
" [OK] I (NOII ): cols=['stock_locate', 'tracking_number', 'timestamp', 'paired_shares']...\n",
|
||
" [OK] J (LULD Auction Collar ): cols=['stock_locate', 'tracking_number', 'timestamp', 'stock']...\n",
|
||
" [OK] K (IPO Quoting Period ): cols=['stock_locate', 'tracking_number', 'timestamp', 'stock']...\n",
|
||
" [OK] L (Market Participant Position): cols=['stock_locate', 'tracking_number', 'timestamp', 'mpid']...\n",
|
||
" [OK] P (Trade ): cols=['stock_locate', 'tracking_number', 'timestamp', 'order_reference_number']...\n"
|
||
]
|
||
},
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
" [OK] Q (Cross Trade ): cols=['stock_locate', 'tracking_number', 'timestamp', 'shares']...\n",
|
||
" [OK] R (Stock Directory ): cols=['stock_locate', 'tracking_number', 'timestamp', 'stock']...\n",
|
||
" [OK] S (System Event ): cols=['stock_locate', 'tracking_number', 'timestamp', 'event_code']...\n"
|
||
]
|
||
},
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
" [OK] U (Order Replace ): cols=['stock_locate', 'tracking_number', 'timestamp', 'original_order_reference_number']...\n",
|
||
" [OK] V (MWCB Decline Level ): cols=['stock_locate', 'tracking_number', 'timestamp', 'level_1']...\n",
|
||
" [OK] X (Order Cancel ): cols=['stock_locate', 'tracking_number', 'timestamp', 'order_reference_number']...\n",
|
||
" [OK] Y (Reg SHO Restriction ): cols=['stock_locate', 'tracking_number', 'timestamp', 'stock']...\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Schema compatibility check — verify we can read each message type\n",
|
||
"print(\"Schema Compatibility Check:\")\n",
|
||
"print(\"-\" * 50)\n",
|
||
"for msg_dir in sorted(MESSAGE_DIR.iterdir()):\n",
|
||
" if (\n",
|
||
" msg_dir.is_dir()\n",
|
||
" and len(msg_dir.name) == 1\n",
|
||
" and msg_dir.name.isupper()\n",
|
||
" and list(msg_dir.glob(\"*.parquet\"))\n",
|
||
" ):\n",
|
||
" try:\n",
|
||
" sample = pl.scan_parquet(msg_dir / \"*.parquet\").head(5).collect()\n",
|
||
" name = MESSAGE_SPECS.get(msg_dir.name, {}).get(\"name\", \"Unknown\")\n",
|
||
" print(f\" [OK] {msg_dir.name} ({name:25}): cols={list(sample.columns)[:4]}...\")\n",
|
||
" except Exception as e:\n",
|
||
" print(f\" [FAIL] {msg_dir.name}: {e}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "1cfeb0a5",
|
||
"metadata": {
|
||
"tags": []
|
||
},
|
||
"source": [
|
||
"## 6. Production Parsing with Rust\n",
|
||
"\n",
|
||
"The Python parser above is educational but slow for full-day files.\n",
|
||
"For production use, we provide a **Rust parser** that is an order of magnitude faster.\n",
|
||
"\n",
|
||
"**Repository**: [github.com/ml4t/itch-parser](https://github.com/ml4t/itch-parser)\n",
|
||
"\n",
|
||
"### Performance Characteristics\n",
|
||
"\n",
|
||
"| Aspect | Python | Rust |\n",
|
||
"|--------|--------|------|\n",
|
||
"| Speed | Baseline | **10-20× faster** |\n",
|
||
"| Memory | High (buffers in RAM) | Low (streaming) |\n",
|
||
"| Use case | Learning, debugging | Production pipelines |\n",
|
||
"\n",
|
||
"Actual speedups depend on disk I/O and CPU. The Rust parser uses memory-mapped\n",
|
||
"I/O and zero-copy parsing, which provides substantial gains on modern hardware.\n",
|
||
"\n",
|
||
"### Installation\n",
|
||
"\n",
|
||
"```bash\n",
|
||
"# Clone the repository\n",
|
||
"git clone https://github.com/ml4t/itch-parser.git\n",
|
||
"cd itch-parser\n",
|
||
"\n",
|
||
"# Build release binary\n",
|
||
"cargo build --release\n",
|
||
"```\n",
|
||
"\n",
|
||
"### Usage\n",
|
||
"\n",
|
||
"```bash\n",
|
||
"# Parse ITCH file (works with .gz or uncompressed)\n",
|
||
"./target/release/itch_parser <input_file> <output_dir> <MMDDYYYY>\n",
|
||
"\n",
|
||
"# Example\n",
|
||
"./target/release/itch_parser data/01302020.NASDAQ_ITCH50.gz ./messages 01302020\n",
|
||
"```\n",
|
||
"\n",
|
||
"Output is identical Parquet files partitioned by message type, compatible with\n",
|
||
"the Python code in this notebook and downstream analysis.\n",
|
||
"\n",
|
||
"### When to Use Which\n",
|
||
"\n",
|
||
"| Use Case | Recommendation |\n",
|
||
"|----------|---------------|\n",
|
||
"| Learning the protocol | Python (this notebook) |\n",
|
||
"| Debugging parse issues | Python |\n",
|
||
"| Processing a single day | Either |\n",
|
||
"| Multi-day backtesting | **Rust** |\n",
|
||
"| Production pipeline | **Rust** |"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "3cc4a0f3",
|
||
"metadata": {
|
||
"tags": []
|
||
},
|
||
"source": [
|
||
"## Key Takeaways\n",
|
||
"\n",
|
||
"1. **ITCH Protocol**: Binary message-by-order format with nanosecond precision\n",
|
||
"2. **Message Types**: A/F (add), E/C (execute), X (cancel), D (delete), U (replace)\n",
|
||
"3. **Price Format**: Integers with 4 implied decimals (150.0000 → 1500000)\n",
|
||
"4. **Parser Choice**: Python for learning, Rust for production (20× faster)\n",
|
||
"\n",
|
||
"### Next Steps\n",
|
||
"\n",
|
||
"- **Order Book Reconstruction**: `02_itch_lob_reconstruction`\n",
|
||
"- **Trading Activity Overview**: `05_itch_trading_activity` (includes E/C enrichment)\n",
|
||
"\n",
|
||
"---\n",
|
||
"\n",
|
||
"## Reference\n",
|
||
"\n",
|
||
"Bouchaud, J.-P., Bonart, J., Donier, J., & Gould, M. (2018).\n",
|
||
"*Trades, Quotes and Prices: Financial Markets Under the Microscope*.\n",
|
||
"Cambridge University Press.\n",
|
||
"[https://doi.org/10.1017/9781009028943](https://doi.org/10.1017/9781009028943)"
|
||
]
|
||
}
|
||
],
|
||
"metadata": {
|
||
"jupytext": {
|
||
"formats": "ipynb,py:percent"
|
||
},
|
||
"kernelspec": {
|
||
"display_name": "Python 3 (ipykernel)",
|
||
"language": "python",
|
||
"name": "python3"
|
||
},
|
||
"language_info": {
|
||
"codemirror_mode": {
|
||
"name": "ipython",
|
||
"version": 3
|
||
},
|
||
"file_extension": ".py",
|
||
"mimetype": "text/x-python",
|
||
"name": "python",
|
||
"nbconvert_exporter": "python",
|
||
"pygments_lexer": "ipython3",
|
||
"version": "3.14.3"
|
||
},
|
||
"papermill": {
|
||
"default_parameters": {},
|
||
"duration": 3.216159,
|
||
"end_time": "2026-05-15T19:16:23.704747+00:00",
|
||
"environment_variables": {},
|
||
"exception": null,
|
||
"input_path": "03_market_microstructure/01_itch_parser.ipynb",
|
||
"output_path": "03_market_microstructure/01_itch_parser.ipynb",
|
||
"parameters": {
|
||
"SKIP_PARSING": true
|
||
},
|
||
"start_time": "2026-05-15T19:16:20.488588+00:00",
|
||
"version": "2.7.0"
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 5
|
||
}
|