Chapter 2: The Financial Data Universe

The chapter gives readers the conceptual map they need before touching any dataset. Its key contribution is not just the market / fundamental / alternative taxonomy, but the claim that every dataset embeds definitions about timestamps, adjustments, identifiers, and revisions, and that these choices determine what the data actually means in research.

Learning Objectives

Distinguish among market, fundamental, and alternative data, and explain how dataset definitions shape what each source means in research and trading applications
Compare the observability, conventions, and engineering constraints of major asset classes, and identify how market structure changes what can be measured and modeled
Apply a financial data quality framework to diagnose common failure modes, especially point-in-time violations, survivorship bias, corporate action errors, and identifier mismatches
Conduct vendor due diligence across data quality, legal and compliance, and technical and commercial dimensions
Choose storage and query architectures that fit research and production needs, including when to use partitioned files, embedded analytical databases, or server-based systems

Sections

2.1 A Modern Taxonomy of Financial Data

This section gives readers the conceptual map they need before touching any dataset. Its key contribution is not just the market / fundamental / alternative taxonomy, but the claim that every dataset embeds definitions about timestamps, adjustments, identifiers, and revisions, and that these choices determine what the data actually means in research.

2.2 The Asset-Class Market Data Landscape

This section broadens the discussion from data categories to the practical reality that "price," "liquidity," and even "the dataset" mean different things across equities, ETPs, futures, options, digital assets, FX, fixed income, swaps, and commodities. Its value is comparative: it helps readers understand why engineering choices are inseparable from market structure.

01_us_equities_eda — This notebook introduces the Wiki Prices dataset - a survivorship-bias-free collection of US equity prices. Understanding survivorship bias is critical for realistic backtesting.
02_corporate_actions — This notebook demonstrates how stock splits and dividends break historical price series, and shows the industry-standard backward adjustment methodology used by major data vendors. Correctly adjusting for corporate actions is essential for any ML model using return-based features.
03_etfs_eda — This notebook introduces the 50-ETF universe that serves as the foundation for the ETF Rotational Momentum case study throughout the book. We explore the schema, coverage, categories, and data quality characteristics.
04_cme_futures_eda — This notebook introduces the CME futures dataset shipped with the book. It demonstrates the data structure, coverage, and key concepts for working with futures data.
05_futures_session_aggregation — This notebook converts hourly continuous futures data (stored in UTC) to session-aware daily bars. CME futures sessions end at 4:00 PM Central Time, so daily bars must respect this boundary—not midnight UTC.
06_futures_continuous — This notebook tackles one of the most critical challenges in futures analysis: creating continuous price series from individual expiring contracts. We implement roll detection algorithms and adjustment methods (Panama, ratio) to eliminate artificial price gaps while preserving accurate return characteristics.
07_sp500_options_eda — This notebook provides a comprehensive exploration of the AlgoSeek S&P 500 Options Analytics dataset. Options data is fundamentally different from spot market data—it contains forward-looking information about expected volatility, directional sentiment, and tail risk that isn't directly observable in underlying prices.
08_options_greeks_computation — This notebook derives and implements the Black-Scholes option pricing framework from first principles. We compute implied volatility and all Greeks, then validate our calculations against the pre-computed values in the AlgoSeek options data.
09_options_continuous — Options are time-decaying instruments. Unlike equities or futures, an option's price reflects both the value of the underlying exposure and the remaining time to expiration.
10_crypto_perps_eda — This notebook introduces the cryptocurrency dataset from Binance Futures. We explore hourly OHLCV data and the Premium Index (perpetual futures vs spot spread) that forms the basis for the Crypto Premium Arbitrage case study.
11_crypto_premium_analysis — This notebook demonstrates how to work with Binance perpetual futures premium index data - the foundation for funding rate arbitrage strategies. We load, explore, and analyze premium dynamics across major cryptocurrencies to identify potential arbitrage opportunities.
12_fx_pairs_eda — This notebook introduces the FX dataset from OANDA. FX markets are OTC with no centralized exchange, so prices aggregate from multiple liquidity providers.

2.3 Data Sourcing: The Due Diligence Framework

This is the chapter's risk-control core. It explains that many apparent research successes are manufactured by data defects, then organizes due diligence around general quality dimensions, finance-specific failure modes, vendor evaluation, and internal governance. The section turns abstract warnings into operational rules: point-in-time correctness, survivorship handling, corporate action methodology, identifier integrity, legal rights, and reproducible versioning.

13_data_quality_framework — The ml4t-data library provides purpose-built tools for financial data quality. This notebook demonstrates the complete data quality workflow: Uses us_equities data.
14_point_in_time_validation — Point-in-time correctness is essential for valid backtesting. Using information that wasn't available at decision time creates lookahead bias - making backtests look better than they would perform in live trading.
15_survivorship_bias_detection — Survivorship bias is arguably the most dangerous form of data contamination in quantitative finance. This notebook uses real historical data from the US equities dataset (US Equities, originally Quandl WIKI) to demonstrate, detect, and quantify survivorship bias.
16_provider_comparison — ML4T Third Edition - Chapter 2: The Financial Data...

2.4 Data Storage

This section translates data discipline into infrastructure decisions. Rather than promoting a single stack, it explains how storage choice depends on access patterns, scale, concurrency, and operational maturity, and it benchmarks the trade-offs among file formats, embedded engines, and server databases. It gives readers a practical default architecture for modern research workflows while also teaching when more complex systems are justified.

17_complete_pipeline — This notebook demonstrates end-to-end data pipelines, bringing together concepts from this chapter: Uses crypto_perps, wiki_provider data.
18_data_management — Previous notebooks fetched and validated data. This notebook shows how to manage it at scale using ml4t-data's production features: Uses universe data.
19_incremental_updates — The previous notebook introduced DataManager and HiveStorage. This notebook focuses on the update workflow — the core reason ml4t-data exists: Uses all, treasury_yields data.
20_storage_benchmark_file — Focus: Pure file format comparison (no query engines) Technologies: CSV, Parquet, Feather (Arrow IPC), HDF5 Operations: Write, Read (with forced materialization), Columnar...
21_storage_benchmark_database — > Docker required: This notebook depends on the benchmark environment and > database services.
22_pandas_polars_benchmark — DataFrame-engine comparison (pandas vs Polars) across read, filter, groupby, join, and lazy operations on synthetic financial data at S/M/L scales. Backs the in-memory engine-choice recommendation in §2.4.

Running the Notebooks

# From the repository root
uv run python 02_financial_data_universe/<notebook>.py

# Test mode (reduced data via Papermill)
uv run pytest tests/test_notebooks.py -v -k "02_financial_data_universe"

NB21 storage_benchmark_database prerequisites

21_storage_benchmark_database exercises 7 storage engines (DuckDB, SQLite, TimescaleDB, ClickHouse, QuestDB, PostgreSQL, Polars Parquet). Of these, DuckDB and SQLite run locally with no additional setup; the other five require the benchmark service stack to be running first:

docker compose --profile benchmark up -d

Without these services, NB21 silently falls back to the 2-engine subset (DuckDB + SQLite only) and the comparative server-database results in §2.4 will not be reproduced. Bring the stack down with docker compose --profile benchmark down when finished.

References

William Beaver et al. (2007). Delisting returns and their effect on accounting-based market anomalies. Journal of Accounting and Economics.
Florian Berg et al. (2022). Aggregate Confusion: The Divergence of ESG Ratings*. Review of Finance.
Mark M. Carhart et al. (2002). Mutual Fund Survivorship. The Review of Financial Studies.
Lin William Cong et al. (2023). Crypto Wash Trading. Management Science.
David Easley et al. (2021). Microstructure in the Machine Age. The Review of Financial Studies.
B. Espen Eckbo and Markus Lithell (2025). Merger-Driven Listing Dynamics. Journal of Financial and Quantitative Analysis.
Gene Ekster and Petter N. Kolm (2020). Alternative Data in Investment Management: Usage, Challenges and Valuation.
Edwin J. Elton et al. (1996). Survivor Bias and Mutual Fund Performance. The Review of Financial Studies.
Kingsley Y L Fong et al. (2017). What Are the Best Liquidity Proxies for Global Research?*. Review of Finance.
Songrun He et al. (2024). Fundamentals of Perpetual Futures.
Jacques Joubert et al. (2024). The Three Types of Backtests.
Nina Karnaukh et al. (2015). Understanding FX Liquidity.
Gueorgui S. Konstantinov (2025). On Systematic Currency Management. The Journal of Portfolio Management.
John Lehoczky and Mark Schervish (2018). Overview and History of Statistics for Equity Markets. Annual Review of Statistics and Its Application.
Alex Lipton and Marcos Lopez de Prado (2020). Three Quant Lessons from COVID-19.
Tim Loughran and Bill McDonald (2020). Textual Analysis in Finance. Annual Review of Financial Economics.
Yin Luo et al. (2014). Seven Sins of Quantitative Investing.
Igor Makarov and Antoinette Schoar (2020). Trading and arbitrage in cryptocurrency markets. Journal of Financial Economics.
Hunter Ng et al. (2025). Price Discovery and Trading in Prediction Markets.
Maureen O’Hara (2015). High frequency market microstructure. Journal of Financial Economics.
Marcos Lopez de Prado (2018). Advances in Financial Machine Learning. John Wiley & Sons.
SEC (2020). Staff Report on Algorithmic Trading in U.S. Capital Markets.
Tyler Shumway (1997). The Delisting Bias in CRSP Data. The Journal of Finance.
David Vidal-Tomás (2022). Which cryptocurrency data sources should scholars use?. International Review of Financial Analysis.

13 KiB Raw Permalink Blame History Unescape Escape