# ---
# jupyter:
#   jupytext:
#     cell_metadata_filter: tags,-all
#     text_representation:
#       extension: .py
#       format_name: percent
#       format_version: '1.3'
#       jupytext_version: 1.19.3
#   kernelspec:
#     display_name: Python 3 (ipykernel)
#     language: python
#     name: python3
# ---

# %% [markdown]
# # FX Pairs — Exploratory Data Analysis
#
# **Docker image**: `ml4t`
#
# ## Purpose
# Profile the OANDA 20-pair, 4-hour FX dataset that anchors the FX case study.
# FX is OTC: there is no central tape, so quotes and reported volumes are
# venue-specific. The notebook surveys coverage, quote conventions, OHLC
# integrity, and the 4h→daily aggregation used by downstream chapters.
#
# ## Learning Objectives
# - Load and inspect the 4-hour OHLC + indicative-volume panel for 20 pairs.
# - Distinguish direct (USD-quoted), indirect (USD-base), and cross pairs.
# - Read FX volume as an OANDA indicator, not an authoritative tape.
# - Aggregate 4h bars to UTC-day daily bars and read the gap-pattern signal.
#
# ## Book reference
# Chapter 2, §2.2 (asset-class market data — foreign exchange). The FX case
# study built on this dataset lives in `case_studies/fx_pairs/`.
#
# ## Prerequisites
# - OANDA 4h FX parquet files materialized under `ML4T_DATA_PATH`.
# - Loader `data.load_fx_pairs`.

# %%
"""FX Pairs — Exploratory data analysis of OANDA currency pair data."""

import polars as pl

from data import load_fx_pairs
from utils.data_quality import check_ohlc_invariants, per_asset_stats

# %% tags=["parameters"]
# Production defaults — Papermill injects overrides for CI
# (No tunable knobs: this notebook EDAs the full 12-pair universe via the
# canonical load_fx_pairs() API; there is no MAX_SYMBOLS / START_DATE knob to expose.)

# %% [markdown]
# ## 1. Load and Inspect

# %%
fx_4h = load_fx_pairs(frequency="4h")

print("=== FX Dataset ===")
print(f"Shape: {fx_4h.shape}")
print(f"Columns: {fx_4h.columns}")
print(f"Date range: {fx_4h['timestamp'].min()} to {fx_4h['timestamp'].max()}")

# %% [markdown]
# ### Volume Disclaimer
#
# **Important**: FX is an OTC market. Volume figures are indicative estimates from OANDA,
# not authoritative exchange data. Do not interpret FX volume the same way as equity volume.

# %%
fx_4h.head()

# %%
# Available pairs
pairs = fx_4h["symbol"].unique().sort().to_list()
print(f"\nCurrency pairs ({len(pairs)}):")
for pair in pairs:
    print(f"  {pair}")

# %% [markdown]
# ### Symbol Normalization
#
# The data uses underscore format (`EUR_USD`). The canonical format for this dataset is
# concatenated (`EURUSD`). Here's how to convert:

# %%
# Normalize symbols: EUR_USD → EURUSD (canonical format)
# The raw file uses underscores; we normalize to concatenated format for downstream joins
fx = fx_4h.with_columns(pl.col("symbol").str.replace("_", "").alias("symbol"))

print("Symbol normalization example:")
print("  Raw format: EUR_USD, USD_JPY, GBP_USD")
print("  Canonical:  EURUSD, USDJPY, GBPUSD")
print(f"\nNormalized pairs: {fx['symbol'].unique().sort().to_list()[:6]} ...")

# %% [markdown]
# ## 2. Coverage Summary

# %%
# Per-pair statistics (using normalized symbols)
pairs = fx["symbol"].unique().sort().to_list()

pair_stats = per_asset_stats(
    fx,
    time_col="timestamp",
    asset_col="symbol",
    price_col="close",
    volume_col="volume",
)

pair_stats.sort("avg_volume", descending=True)

# %% [markdown]
# ## 3. Quote Conventions
#
# FX pairs follow **BASE/QUOTE** convention:
#
# | Pair | Interpretation | USD Strength |
# |------|----------------|--------------|
# | EUR/USD = 1.10 | 1 EUR costs 1.10 USD | Down = USD stronger |
# | USD/JPY = 150 | 1 USD costs 150 JPY | Up = USD stronger |
# | EUR/GBP = 0.86 | 1 EUR costs 0.86 GBP | Cross rate (no USD) |
#
# To create a consistent "USD index", you must **invert** EUR/USD and GBP/USD.

# %%
# Quote convention classification (using canonical symbols).
QUOTE_CONVENTIONS = {
    "EURUSD": ("Direct", "USD per EUR", "invert for USD strength"),
    "GBPUSD": ("Direct", "USD per GBP", "invert for USD strength"),
    "AUDUSD": ("Direct", "USD per AUD", "invert for USD strength"),
    "NZDUSD": ("Direct", "USD per NZD", "invert for USD strength"),
    "USDJPY": ("Indirect", "JPY per USD", "direct USD strength"),
    "USDCHF": ("Indirect", "CHF per USD", "direct USD strength"),
    "USDCAD": ("Indirect", "CAD per USD", "direct USD strength"),
    "EURGBP": ("Cross", "GBP per EUR", "no USD"),
    "EURJPY": ("Cross", "JPY per EUR", "no USD"),
}

quote_conventions = pl.DataFrame(
    [
        {"symbol": p, "convention": c, "meaning": m, "usd_strength": n}
        for p, (c, m, n) in QUOTE_CONVENTIONS.items()
        if p in pairs
    ]
)
quote_conventions

# %% [markdown]
# ## 4. Data Quality

# %%
# OHLC invariants
invariants = check_ohlc_invariants(fx)
invariants

# %%
# Check for weekend gaps (expected in FX, which trades 24/5)
eurusd = fx.filter(pl.col("symbol") == "EURUSD").sort("timestamp")

eurusd_gaps = eurusd.with_columns(
    pl.col("timestamp").diff().dt.total_hours().alias("hours_since_prev")
)

large_gaps = eurusd_gaps.filter(pl.col("hours_since_prev") > 24)
print(f"\nGaps > 24 hours (EURUSD): {len(large_gaps)} (should be weekends only)")

# %% [markdown]
# ## 5. Daily Aggregation
#
# Aggregate 4-hour bars to daily for consistency with other datasets.
#
# **Note**: This uses UTC midnight boundaries. FX daily bars are conventionally defined
# by a session cutoff (often 5pm New York). For production, align to your broker's
# convention. This simple calendar-day aggregation is sufficient for exploration.

# %%
# Daily aggregation (must be sorted for group_by_dynamic)
fx_daily = (
    fx.sort("symbol", "timestamp")
    .group_by_dynamic("timestamp", every="1d", group_by="symbol")
    .agg(
        [
            pl.col("open").first(),
            pl.col("high").max(),
            pl.col("low").min(),
            pl.col("close").last(),
            pl.col("volume").sum(),
        ]
    )
)

print(f"Daily aggregation (UTC boundaries) — shape: {fx_daily.shape}")
fx_daily.filter(pl.col("symbol") == "EURUSD").tail(5)

# %% [markdown]
# ## Key Takeaways
#
# Profile of the OANDA 4h FX panel that anchors the FX case study.
#
# ### Quantitative Findings
# - **Panel scale**: 478,640 4h observations across 20 currency pairs spanning
#   2011-01-02 → 2025-12-31. Each pair has ~23,920–23,945 4h bars.
# - **Liquidity tiers (by indicative OANDA volume)**: GBPAUD, GBPJPY, EURCAD,
#   GBPCHF and CHFJPY top the table at 18k–28k contracts/4h; the bottom of
#   the universe (NZDUSD, EURCHF, EURGBP, AUDUSD, USDCHF) sits at 5k–7k.
#   These rankings are *OANDA-specific* — interbank-market liquidity for
#   EURUSD/USDJPY is the largest globally but is not visible to a single
#   retail venue.
# - **OHLC integrity**: 100% of 4h bars satisfy all six invariants
#   (high ≥ low/open/close, low ≤ open/close, volume ≥ 0).
# - **Session gaps**: 747 EURUSD inter-bar gaps exceed 24 h, matching the
#   ~770 weekend closes over 14 years — confirming the 24/5 calendar.
# - **Daily roll-up**: UTC-boundary aggregation produces 94,642 daily rows
#   across the panel (≈4,732 trading days × 20 pairs).
#
# ### Implications for Practitioners
# - **Volume**: Treat as a relative liquidity indicator across pairs on this
#   venue, not as an interbank tape.
# - **Quote inversion**: USD strength composites must invert direct pairs
#   (EUR/USD, GBP/USD, AUD/USD, NZD/USD); USD-base and cross pairs do not
#   need inversion.
# - **Daily session convention**: UTC-day aggregation is convenient for joins
#   with the equity/crypto panels but is *not* a tradable session boundary;
#   broker-specific 5pm-NY cutoffs are wired in `case_studies/fx_pairs/`
#   downstream.
#
# **Next**: `13_data_quality_framework` profiles the cross-asset DQ checks
# that consume this panel and the others built up so far.