The chapter establishes the chapter's central claim: in trading, durable performance depends less on picking a sophisticated model than on maintaining a disciplined research process that can survive changing markets, noisy signals, and real-world frictions. It gives readers a usable vocabulary for market change, shows why recent shocks exposed fragile assumptions, and reframes ML for trading as an adaptation problem rather than a model-selection contest.

Learning Objectives

Distinguish structural breaks, regimes, data drift, concept drift, and online detection, and explain why static trading models degrade in changing markets
Explain the ML4T Workflow as a research-to-production system, including its data infrastructure foundation, scoping invariants, iterative research modules, and feedback loops from live trading back to research
Define the evidence boundary between exploration and confirmation, and explain how trial logging, sealed holdouts, and selection-aware evaluation preserve research integrity
Describe how causal inference and generative AI fit within a disciplined trading workflow, including the main benefits they provide and the new failure modes they introduce
Apply regime thinking, implementability checks, and monitoring logic to diagnose strategy vulnerabilities and to adapt workflow discipline across independent and institutional settings

Sections

1.1 Why Process Discipline Matters

This section establishes the chapter's central claim: in trading, durable performance depends less on picking a sophisticated model than on maintaining a disciplined research process that can survive changing markets, noisy signals, and real-world frictions. It gives readers a usable vocabulary for market change, shows why recent shocks exposed fragile assumptions, and reframes ML for trading as an adaptation problem rather than a model-selection contest.

1.2 Introducing the ML4T Workflow

This section presents the book's core framework: a research-to-production workflow built on point-in-time-correct data infrastructure, explicit scoping rules, iterative feature and model development, realistic strategy design, deployment discipline, and ongoing monitoring. The key value for readers is that it turns trading research into a managed lifecycle with auditable artifacts, clear handoffs, and an explicit boundary between exploration and confirmation.

1.3 Causal Inference and Generative AI in the Workflow

This section places two modern method families inside the workflow rather than treating them as standalone trends. Causal inference is framed as a way to sharpen mechanisms, assumptions, and diagnosis; generative AI is framed as a way to expand research and unstructured-data processing while also creating new risks such as leakage, hallucination, and workflow bloat. Readers should care because the section makes clear that new tools increase the value of discipline rather than replacing it.

1.4 Market Regimes: Change Is the Constant

This section turns non-stationarity into something operational. It shows how regime concepts can support explanation, robustness checks, and live monitoring, while insisting that regimes are primarily a risk lens rather than a reliable timing signal. The factor and macro examples make the idea concrete: regime methods are useful when they help identify adverse environments and connect them to predefined risk actions.

factor_regimes — Demonstrates unsupervised learning for market regime detection using Gaussian Mixture Models (GMM) on factor returns from the AQR Century of Factor Premia dataset.
macro_regimes — Demonstrates unsupervised learning for market regime detection using macroeconomic indicators from FRED, validated against S&P 500 volatility and drawdowns.

1.5 In the Real World: Independent vs. Institutional

This section translates the workflow into real operating contexts. It explains how institutions benefit from built-in friction and review, while independent researchers must create their own governance through documentation, checkpoints, and explicit stop criteria. The practical payoff is strong: it helps readers see where solo practitioners are vulnerable, where they can still compete, and how reusable infrastructure compounds research quality over time.

Running the Notebooks

# From the repository root
uv run python 01_process_is_edge/<notebook>.py

# Test mode (reduced data via Papermill)
uv run pytest tests/test_notebooks.py -v -k "01_process_is_edge"

References

Andrew Ang and Geert Bekaert (2002). International Asset Allocation With Regime Shifts. Review of Financial Studies.
Robert D. Arnott et al. (2018). A Backtesting Protocol in the Era of Machine Learning.
Darrell Duffie (2020). Still the World's Safe Haven? Redesigning the U.S. Treasury Market After the COVID-19 Crisis.
David Easley et al. (2012). The Volume Clock: Insights into the High Frequency Paradigm.
Frank J. Fabozzi et al. (2024). Paradigm Shift: Embracing Holism in Causal Modeling for Investment Applications. The Journal of Portfolio Management.
Frank J. Fabozzi and Caleb C. Stenholm (2025). Strategic Discipline: How Asset Management Mirrors Military Operations. The Journal of Portfolio Management.
Ziang Fang and Jason Moore (2025). What AI Can (and Can't Yet) Do for Alpha.
Stefano Giglio et al. (2022). Factor Models, Machine Learning, and Asset Pricing. Annual Review of Financial Economics.
Campbell R. Harvey et al. (2016). ...and the Cross-Section of Expected Returns. Review of Financial Studies.
Blanka Horvath et al. (2021). Clustering Market Regimes Using the Wasserstein Distance.
Antti Ilmanen et al. (2021). How Do Factor Premia Vary Over Time? A Century of Evidence.
Justina Lee (2025). Man Group Says Agentic AI Is Now Devising Quant Trading Signals. Bloomberg.com.
Andrew W. Lo (2004). The Adaptive Markets Hypothesis: Market Efficiency from an Evolutionary Perspective.
Martin Luk (2023). Generative AI: Overview, Economic Impact, and Applications in Asset Management.
Judea Pearl (2019). The seven tools of causal inference, with reflections on machine learning. Communications of the ACM.
Marcos López de Prado (2018). The 10 Reasons Most Machine Learning Funds Fail. The Journal of Portfolio Management.
Marcos Lopez de Prado et al. (2024). The Case for Causal Factor Investing.
Marcos López de Prado and Vincent Zoonekynd (2025). Correcting the Factor Mirage: A Research Protocol for Causal Factor Investing. The Journal of Portfolio Management.
James Ryseff et al. (2024). The Root Causes of Failure for Artificial Intelligence Projects and How They Can Succeed: Avoiding the Anti-Patterns of AI.
Bernhard Schölkopf et al. (2021). Towards Causal Representation Learning.
Stefan Studer et al. (2021). Towards CRISP-ML(Q): A Machine Learning Process Model with Quality Assurance Methodology. Machine Learning and Knowledge Extraction.
A. Sinem Uysal and John M. Mulvey (2021). A Machine Learning Approach in Regime-Switching Risk Parity Portfolios. The Journal of Financial Data Science.