wehub-resource-sync
113 lines
13 KiB
Markdown
113 lines
13 KiB
Markdown
# Chapter 3: Market Microstructure
|
||
|
||
The chapter explains why market data cannot be treated as a neutral price series. It shows how spreads, depth, resiliency, order types, and intraday trading regimes shape both execution quality and the meaning of observed trades and quotes. For readers building trading systems, this is the section that turns "price data" into an economic object with frictions, incentives, and timing effects.
|
||
|
||
## Learning Objectives
|
||
|
||
* Explain how liquidity, order types, market design, and intraday trading regimes shape observed market data and execution quality.
|
||
* Distinguish among major market data products, including L1, L2, L3, TAQ, and enriched bar datasets, and choose data that matches a research or trading objective.
|
||
* Parse message-based exchange data and reconstruct a venue-local limit order book while enforcing core lifecycle and accounting invariants.
|
||
* Interpret key order-book measures and empirical microstructure patterns, while recognizing the limits of visible single-venue data.
|
||
* Build and compare time-, activity-, and information-driven bars, including when trade-direction classification and Lee-Ready alignment are required.
|
||
* Apply intraday data-quality and sessionization checks that prevent sequencing, timestamp, and calendar errors from contaminating downstream analysis.
|
||
|
||
## Sections
|
||
|
||
### 3.1 Microstructure: The DNA of Price Formation
|
||
|
||
This section explains why market data cannot be treated as a neutral price series. It shows how spreads, depth, resiliency, order types, and intraday trading regimes shape both execution quality and the meaning of observed trades and quotes. For readers building trading systems, this is the section that turns "price data" into an economic object with frictions, incentives, and timing effects.
|
||
|
||
### 3.2 The Anatomy of Modern Market Data Feeds
|
||
|
||
This section maps the market data hierarchy from top-of-book quotes to full order-level feeds and makes clear that data choice is a strategy choice. It helps readers understand what L1, L2, L3, TAQ, proprietary feeds, and enriched vendor products can and cannot reveal, and why timestamp semantics, latency, and fragmentation affect both research validity and live feasibility.
|
||
|
||
### 3.3 From Raw Messages to the Limit Order Book
|
||
|
||
This section turns raw exchange messages into a concrete engineering workflow: parse, normalize, replay, validate, and analyze a venue-local order book. It matters because it connects message traffic to state, shows how to maintain book invariants, and clarifies both the power and the limits of reconstructed visible liquidity. It also gives the chapter empirical weight by tying reconstruction to concrete stylized facts and practical research outputs.
|
||
|
||
- [`01_itch_parser`](01_itch_parser.ipynb) — This notebook demonstrates how to parse NASDAQ's TotalView-ITCH binary protocol. Understanding MBO (message-by-order) data is foundational for microstructure-based ML features. (_runtime: ~22 minutes; ~8 GB RAM_)
|
||
- [`02_itch_lob_reconstruction`](02_itch_lob_reconstruction.ipynb) — This notebook reconstructs the limit order book (LOB) from ITCH messages. Uses itch_messages data.
|
||
- [`03_itch_lob_analysis`](03_itch_lob_analysis.ipynb) — This notebook analyzes reconstructed order book data to demonstrate empirically-grounded patterns with predictive power. These patterns motivate feature engineering in Chapter 7 and execution cost modeling in Chapter 19. (_~13 GB RAM_)
|
||
- [`04_itch_order_lifecycle_analysis`](04_itch_order_lifecycle_analysis.ipynb) — This notebook analyzes the complete lifecycle of limit orders using NASDAQ ITCH data, revealing critical insights about cancellation rates, time dynamics, and the role of high-frequency market making in modern markets. Uses message_type, nasdaq_itch data. (_~33 GB RAM_)
|
||
- [`05_itch_trading_activity`](05_itch_trading_activity.ipynb) — This notebook provides a high-level view of NASDAQ trading activity using TotalView-ITCH data. We examine message composition and volume concentration across securities. (_~31 GB RAM_)
|
||
- [`06_itch_intraday_patterns`](06_itch_intraday_patterns.ipynb) — This notebook analyzes intraday volume patterns from NASDAQ ITCH data, demonstrating the well-documented U-shape. Volatility patterns are covered in 07_itch_stylized_facts.py (spread dynamics, bid-ask bounce).
|
||
- [`07_itch_stylized_facts`](07_itch_stylized_facts.ipynb) — This notebook demonstrates fundamental microstructure patterns: order arrival/cancellation dynamics, the bid-ask bounce phenomenon, and the liquidity spectrum across stocks. Uses add_cancel_for_ticker data.
|
||
- [`08_databento_lob_reconstruction`](08_databento_lob_reconstruction.ipynb) — Part 1 of 3: LOB Reconstruction → Order Flow Analysis → Bar Calibration Uses mbo_data data.
|
||
- [`09_databento_mbo_analysis`](09_databento_mbo_analysis.ipynb) — Part 2 of 3: LOB Reconstruction → Order Flow Analysis → Bar Calibration Uses databento_day, mbo_data data.
|
||
- [`10_iex_lob_reconstruction`](10_iex_lob_reconstruction.ipynb) — This notebook demonstrates LOB reconstruction using IEX's free historical market data, providing a reader-runnable alternative to the NASDAQ ITCH examples in earlier notebooks. Uses iex_hist data.
|
||
- [`11_algoseek_taq_eda`](11_algoseek_taq_eda.ipynb) — On March 16, 2020, the S&P 500 fell 12% - its worst single-day drop since 1987. Circuit breakers halted trading for 15 minutes when the index fell 7% at open. (_~22 GB RAM_)
|
||
- [`12_algoseek_taq_lob_reconstruction`](12_algoseek_taq_lob_reconstruction.ipynb) — Every trade has two sides: a buyer and a seller. But one side is aggressive - they crossed the spread to execute immediately. (_~22 GB RAM_)
|
||
- [`13_algoseek_minute_bars_eda`](13_algoseek_minute_bars_eda.ipynb) — This notebook provides a comprehensive exploration of the AlgoSeek TAQ minute bar dataset. Unlike simple OHLCV data, TAQ bars contain 61 pre-computed columns that capture the full microstructure of each trading minute: quote dynamics, trade execution, aggressor behavior, and liquidity conditions.
|
||
|
||
### 3.4 The Art of Sampling: From Ticks to Bars
|
||
|
||
This section argues that bar construction is not bookkeeping but measurement design. It compares time, tick, volume, dollar, and information-driven bars, explains trade classification and Lee-Ready, and gives readers a grounded basis for choosing simpler activity-time bars versus more demanding imbalance-based methods. The payoff is direct relevance for machine learning: the sampling rule changes the statistical properties of the training data.
|
||
|
||
- [`14_itch_bar_sampling`](14_itch_bar_sampling.ipynb) — Traditional OHLCV bars sample data at fixed time intervals. But markets don't generate information at constant rates.
|
||
- [`15_itch_lee_ready`](15_itch_lee_ready.ipynb) — This notebook validates our Lee-Ready implementation against DataBento's ground truth aggressor labels. Uses databento_mbo, mbo_data data.
|
||
- [`16_itch_information_bars`](16_itch_information_bars.ipynb) — This notebook has two parts: Uses mbo_data, trades data.
|
||
- [`17_databento_bar_sampling`](17_databento_bar_sampling.ipynb) — Part 3 of 3: LOB Reconstruction → Order Flow Analysis → Bar Calibration Uses mbo_data, multiday_trades data.
|
||
|
||
### 3.5 Detecting Price Jumps in Intraday Returns
|
||
|
||
This section separates the continuous and jump components of intraday returns using bipower variation and the Lee–Mykland (2008) test. The decomposition matters because realized variance, tail-heavy returns, and label distributions all change shape once jumps are removed, and the resulting jump features feed label conditioning in Chapter 7 and volatility-regime features in Chapter 8.
|
||
|
||
- [`18_algoseek_jump_detection`](18_algoseek_jump_detection.ipynb) — Decomposes daily realized variance into continuous (bipower) and jump components on AlgoSeek minute bars; applies the Lee–Mykland nonparametric test to time individual jumps with a multiple-testing-correct critical value; compares with a naive |z|>4 rule that misses most jumps because a single daily volatility ignores time-of-day shape; emits a per-symbol-day jump-feature panel.
|
||
|
||
### 3.6 Data Quality and Sessionization
|
||
|
||
This section focuses on intraday failure modes that quietly corrupt research: sequencing errors, timestamp confusion, stale quotes, invalid book transitions, bad qualifiers, and session-boundary mistakes. It is especially useful because it frames data quality as invariants and auditability rather than generic cleaning, giving readers a practical QA mindset for microstructure work.
|
||
|
||
## Running the Notebooks
|
||
|
||
```bash
|
||
# From the repository root
|
||
uv run python 03_market_microstructure/<notebook>.py
|
||
|
||
# Test mode (reduced data via Papermill)
|
||
uv run pytest tests/test_notebooks.py -v -k "03_market_microstructure"
|
||
```
|
||
|
||
### Memory requirements
|
||
|
||
The ITCH and AlgoSeek-TAQ notebooks materialize multi-day, multi-symbol message and tick partitions into memory. We recommend at least 32 GB of system RAM (64 GB ideal) when running the chapter end-to-end, since the five high-RAM notebooks below cumulatively exceed 100 GB if executed without releasing memory between runs. Peak resident-set sizes observed on a 64 GB workstation:
|
||
|
||
- `04_itch_order_lifecycle_analysis` — ~33 GB peak
|
||
- `05_itch_trading_activity` — ~31 GB peak
|
||
- `11_algoseek_taq_eda` — ~22 GB peak
|
||
- `12_algoseek_taq_lob_reconstruction` — ~22 GB peak
|
||
- `03_itch_lob_analysis` — ~13 GB peak
|
||
|
||
The remaining numbered notebooks fit comfortably in 8 GB. If your workstation has less than 32 GB, run the high-memory notebooks one at a time with no other heavy processes; restart the kernel between them so Polars releases its thread-local arenas.
|
||
|
||
### Runtime requirements
|
||
|
||
Most notebooks complete in under a minute. The only long-running notebook is the raw ITCH parse:
|
||
|
||
- `01_itch_parser` — ~22 min wall-clock, ~8 GB peak RSS (single-pass parse of one 423 M-message trading day)
|
||
|
||
## References
|
||
|
||
- **Matteo Aquilina et al.** (2021). [Quantifying the High-Frequency Trading “Arms Race”](https://doi.org/10.3386/w29011).
|
||
- **Rama Cont et al.** (2014). [The Price Impact of Order Book Events](https://doi.org/10.1093/jjfinec/nbt003). *Journal of Financial Econometrics*.
|
||
- **David Easley et al.** (2012). [The Volume Clock: Insights into the High Frequency Paradigm](https://doi.org/10.2139/ssrn.2034858).
|
||
- **David Easley et al.** (2021). [Microstructure in the Machine Age](https://doi.org/10.1093/rfs/hhaa078). *The Review of Financial Studies*.
|
||
- **Lawrence R. Glosten and Paul R. Milgrom** (1985). [Bid, ask and transaction prices in a specialist market with heterogeneously informed traders](https://doi.org/10.1016/0304-405X(85)90044-3). *Journal of Financial Economics*.
|
||
- **Martin D. Gould et al.** (2013). [Limit Order Books](http://arxiv.org/abs/1012.0349). *arXiv:1012.0349 [physics, q-fin]*.
|
||
- **Nikolaus Hautsch and Ruihong Huang** (2012). [The market impact of a limit order](https://doi.org/10.1016/j.jedc.2011.09.012). *Journal of Economic Dynamics and Control*.
|
||
- **Craig W. Holden et al.** (2014). [The Empirical Analysis of Liquidity](https://doi.org/10.1561/0500000044). *Foundations and Trends® in Finance*.
|
||
- **Craig W. Holden et al.** (2023). [In the Blink of an Eye: Exchange-to-SIP Latency and Trade Classification Accuracy](https://doi.org/10.2139/ssrn.4441422).
|
||
- **Albert S. Kyle** (1985). [Continuous Auctions and Insider Trading](https://doi.org/10.2307/1913210). *Econometrica*.
|
||
- **Charles M. C. Lee and Mark J. Ready** (1991). [Inferring Trade Direction from Intraday Data](https://doi.org/10.1111/j.1540-6261.1991.tb02683.x). *The Journal of Finance*.
|
||
- **Ananth Madhavan et al.** (1997). [Why Do Security Prices Change? A Transaction-Level Analysis of NYSE Stocks](https://www.jstor.org/stable/2962338). *The Review of Financial Studies*.
|
||
- **Ananth Madhavan** (2002). [Market Microstructure: A Practitioner's Guide](https://www.jstor.org/stable/4480415). *Financial Analysts Journal*.
|
||
- **Andy Novocin and Bruce Weber** (2022). [Emerging Technologies and the Transformation of Exchange Trading Platforms](https://doi.org/10.3905/jpm.2022.1.390). *The Journal of Portfolio Management*.
|
||
- **Maureen O'Hara** (2011). Market Microstructure Theory. *Wiley*.
|
||
- **Maureen O’Hara** (2015). [High frequency market microstructure](https://doi.org/10.1016/j.jfineco.2015.01.003). *Journal of Financial Economics*.
|
||
- **Marcos Lopez de Prado** (2018). Advances in Financial Machine Learning. *John Wiley & Sons*.
|
||
- **Martin Reck** (2022). [Market Design—A Practitioner’s Perspective](https://doi.org/10.3905/jpm.2022.1.383). *The Journal of Portfolio Management*.
|
||
- **SEC** (2020). [Staff Report on Algorithmic Trading in U.S. Capital Markets](https://www.sec.gov/file/staff-report-algorithmic-trading-us-capital-markets).
|
||
- **Robert A. Schwartz et al.** (2022). [Equity Market Structure and the Persistence of Unsolved Problems: A Microstructure Perspective](https://doi.org/10.3905/jpm.2022.1.384). *The Journal of Portfolio Management*.
|
||
- **Zihao Zhang et al.** (2019). [DeepLOB: Deep Convolutional Neural Networks for Limit Order Books](https://doi.org/10.1109/TSP.2019.2907260). *IEEE Transactions on Signal Processing*.
|
||
- **Yichi Zhang et al.** (2025). [ClusterLOB: Enhancing Trading Strategies by Clustering Orders in Limit Order Books](https://doi.org/10.48550/arXiv.2504.20349).
|