wehub-resource-sync
231 lines
7.7 KiB
Python
231 lines
7.7 KiB
Python
# ---
|
||
# jupyter:
|
||
# jupytext:
|
||
# cell_metadata_filter: tags,-all
|
||
# text_representation:
|
||
# extension: .py
|
||
# format_name: percent
|
||
# format_version: '1.3'
|
||
# jupytext_version: 1.19.3
|
||
# kernelspec:
|
||
# display_name: Python 3 (ipykernel)
|
||
# language: python
|
||
# name: python3
|
||
# ---
|
||
|
||
# %% [markdown]
|
||
# # FX Pairs — Exploratory Data Analysis
|
||
#
|
||
# **Docker image**: `ml4t`
|
||
#
|
||
# ## Purpose
|
||
# Profile the OANDA 20-pair, 4-hour FX dataset that anchors the FX case study.
|
||
# FX is OTC: there is no central tape, so quotes and reported volumes are
|
||
# venue-specific. The notebook surveys coverage, quote conventions, OHLC
|
||
# integrity, and the 4h→daily aggregation used by downstream chapters.
|
||
#
|
||
# ## Learning Objectives
|
||
# - Load and inspect the 4-hour OHLC + indicative-volume panel for 20 pairs.
|
||
# - Distinguish direct (USD-quoted), indirect (USD-base), and cross pairs.
|
||
# - Read FX volume as an OANDA indicator, not an authoritative tape.
|
||
# - Aggregate 4h bars to UTC-day daily bars and read the gap-pattern signal.
|
||
#
|
||
# ## Book reference
|
||
# Chapter 2, §2.2 (asset-class market data — foreign exchange). The FX case
|
||
# study built on this dataset lives in `case_studies/fx_pairs/`.
|
||
#
|
||
# ## Prerequisites
|
||
# - OANDA 4h FX parquet files materialized under `ML4T_DATA_PATH`.
|
||
# - Loader `data.load_fx_pairs`.
|
||
|
||
# %%
|
||
"""FX Pairs — Exploratory data analysis of OANDA currency pair data."""
|
||
|
||
import polars as pl
|
||
|
||
from data import load_fx_pairs
|
||
from utils.data_quality import check_ohlc_invariants, per_asset_stats
|
||
|
||
# %% tags=["parameters"]
|
||
# Production defaults — Papermill injects overrides for CI
|
||
# (No tunable knobs: this notebook EDAs the full 12-pair universe via the
|
||
# canonical load_fx_pairs() API; there is no MAX_SYMBOLS / START_DATE knob to expose.)
|
||
|
||
# %% [markdown]
|
||
# ## 1. Load and Inspect
|
||
|
||
# %%
|
||
fx_4h = load_fx_pairs(frequency="4h")
|
||
|
||
print("=== FX Dataset ===")
|
||
print(f"Shape: {fx_4h.shape}")
|
||
print(f"Columns: {fx_4h.columns}")
|
||
print(f"Date range: {fx_4h['timestamp'].min()} to {fx_4h['timestamp'].max()}")
|
||
|
||
# %% [markdown]
|
||
# ### Volume Disclaimer
|
||
#
|
||
# **Important**: FX is an OTC market. Volume figures are indicative estimates from OANDA,
|
||
# not authoritative exchange data. Do not interpret FX volume the same way as equity volume.
|
||
|
||
# %%
|
||
fx_4h.head()
|
||
|
||
# %%
|
||
# Available pairs
|
||
pairs = fx_4h["symbol"].unique().sort().to_list()
|
||
print(f"\nCurrency pairs ({len(pairs)}):")
|
||
for pair in pairs:
|
||
print(f" {pair}")
|
||
|
||
# %% [markdown]
|
||
# ### Symbol Normalization
|
||
#
|
||
# The data uses underscore format (`EUR_USD`). The canonical format for this dataset is
|
||
# concatenated (`EURUSD`). Here's how to convert:
|
||
|
||
# %%
|
||
# Normalize symbols: EUR_USD → EURUSD (canonical format)
|
||
# The raw file uses underscores; we normalize to concatenated format for downstream joins
|
||
fx = fx_4h.with_columns(pl.col("symbol").str.replace("_", "").alias("symbol"))
|
||
|
||
print("Symbol normalization example:")
|
||
print(" Raw format: EUR_USD, USD_JPY, GBP_USD")
|
||
print(" Canonical: EURUSD, USDJPY, GBPUSD")
|
||
print(f"\nNormalized pairs: {fx['symbol'].unique().sort().to_list()[:6]} ...")
|
||
|
||
# %% [markdown]
|
||
# ## 2. Coverage Summary
|
||
|
||
# %%
|
||
# Per-pair statistics (using normalized symbols)
|
||
pairs = fx["symbol"].unique().sort().to_list()
|
||
|
||
pair_stats = per_asset_stats(
|
||
fx,
|
||
time_col="timestamp",
|
||
asset_col="symbol",
|
||
price_col="close",
|
||
volume_col="volume",
|
||
)
|
||
|
||
pair_stats.sort("avg_volume", descending=True)
|
||
|
||
# %% [markdown]
|
||
# ## 3. Quote Conventions
|
||
#
|
||
# FX pairs follow **BASE/QUOTE** convention:
|
||
#
|
||
# | Pair | Interpretation | USD Strength |
|
||
# |------|----------------|--------------|
|
||
# | EUR/USD = 1.10 | 1 EUR costs 1.10 USD | Down = USD stronger |
|
||
# | USD/JPY = 150 | 1 USD costs 150 JPY | Up = USD stronger |
|
||
# | EUR/GBP = 0.86 | 1 EUR costs 0.86 GBP | Cross rate (no USD) |
|
||
#
|
||
# To create a consistent "USD index", you must **invert** EUR/USD and GBP/USD.
|
||
|
||
# %%
|
||
# Quote convention classification (using canonical symbols).
|
||
QUOTE_CONVENTIONS = {
|
||
"EURUSD": ("Direct", "USD per EUR", "invert for USD strength"),
|
||
"GBPUSD": ("Direct", "USD per GBP", "invert for USD strength"),
|
||
"AUDUSD": ("Direct", "USD per AUD", "invert for USD strength"),
|
||
"NZDUSD": ("Direct", "USD per NZD", "invert for USD strength"),
|
||
"USDJPY": ("Indirect", "JPY per USD", "direct USD strength"),
|
||
"USDCHF": ("Indirect", "CHF per USD", "direct USD strength"),
|
||
"USDCAD": ("Indirect", "CAD per USD", "direct USD strength"),
|
||
"EURGBP": ("Cross", "GBP per EUR", "no USD"),
|
||
"EURJPY": ("Cross", "JPY per EUR", "no USD"),
|
||
}
|
||
|
||
quote_conventions = pl.DataFrame(
|
||
[
|
||
{"symbol": p, "convention": c, "meaning": m, "usd_strength": n}
|
||
for p, (c, m, n) in QUOTE_CONVENTIONS.items()
|
||
if p in pairs
|
||
]
|
||
)
|
||
quote_conventions
|
||
|
||
# %% [markdown]
|
||
# ## 4. Data Quality
|
||
|
||
# %%
|
||
# OHLC invariants
|
||
invariants = check_ohlc_invariants(fx)
|
||
invariants
|
||
|
||
# %%
|
||
# Check for weekend gaps (expected in FX, which trades 24/5)
|
||
eurusd = fx.filter(pl.col("symbol") == "EURUSD").sort("timestamp")
|
||
|
||
eurusd_gaps = eurusd.with_columns(
|
||
pl.col("timestamp").diff().dt.total_hours().alias("hours_since_prev")
|
||
)
|
||
|
||
large_gaps = eurusd_gaps.filter(pl.col("hours_since_prev") > 24)
|
||
print(f"\nGaps > 24 hours (EURUSD): {len(large_gaps)} (should be weekends only)")
|
||
|
||
# %% [markdown]
|
||
# ## 5. Daily Aggregation
|
||
#
|
||
# Aggregate 4-hour bars to daily for consistency with other datasets.
|
||
#
|
||
# **Note**: This uses UTC midnight boundaries. FX daily bars are conventionally defined
|
||
# by a session cutoff (often 5pm New York). For production, align to your broker's
|
||
# convention. This simple calendar-day aggregation is sufficient for exploration.
|
||
|
||
# %%
|
||
# Daily aggregation (must be sorted for group_by_dynamic)
|
||
fx_daily = (
|
||
fx.sort("symbol", "timestamp")
|
||
.group_by_dynamic("timestamp", every="1d", group_by="symbol")
|
||
.agg(
|
||
[
|
||
pl.col("open").first(),
|
||
pl.col("high").max(),
|
||
pl.col("low").min(),
|
||
pl.col("close").last(),
|
||
pl.col("volume").sum(),
|
||
]
|
||
)
|
||
)
|
||
|
||
print(f"Daily aggregation (UTC boundaries) — shape: {fx_daily.shape}")
|
||
fx_daily.filter(pl.col("symbol") == "EURUSD").tail(5)
|
||
|
||
# %% [markdown]
|
||
# ## Key Takeaways
|
||
#
|
||
# Profile of the OANDA 4h FX panel that anchors the FX case study.
|
||
#
|
||
# ### Quantitative Findings
|
||
# - **Panel scale**: 478,640 4h observations across 20 currency pairs spanning
|
||
# 2011-01-02 → 2025-12-31. Each pair has ~23,920–23,945 4h bars.
|
||
# - **Liquidity tiers (by indicative OANDA volume)**: GBPAUD, GBPJPY, EURCAD,
|
||
# GBPCHF and CHFJPY top the table at 18k–28k contracts/4h; the bottom of
|
||
# the universe (NZDUSD, EURCHF, EURGBP, AUDUSD, USDCHF) sits at 5k–7k.
|
||
# These rankings are *OANDA-specific* — interbank-market liquidity for
|
||
# EURUSD/USDJPY is the largest globally but is not visible to a single
|
||
# retail venue.
|
||
# - **OHLC integrity**: 100% of 4h bars satisfy all six invariants
|
||
# (high ≥ low/open/close, low ≤ open/close, volume ≥ 0).
|
||
# - **Session gaps**: 747 EURUSD inter-bar gaps exceed 24 h, matching the
|
||
# ~770 weekend closes over 14 years — confirming the 24/5 calendar.
|
||
# - **Daily roll-up**: UTC-boundary aggregation produces 94,642 daily rows
|
||
# across the panel (≈4,732 trading days × 20 pairs).
|
||
#
|
||
# ### Implications for Practitioners
|
||
# - **Volume**: Treat as a relative liquidity indicator across pairs on this
|
||
# venue, not as an interbank tape.
|
||
# - **Quote inversion**: USD strength composites must invert direct pairs
|
||
# (EUR/USD, GBP/USD, AUD/USD, NZD/USD); USD-base and cross pairs do not
|
||
# need inversion.
|
||
# - **Daily session convention**: UTC-day aggregation is convenient for joins
|
||
# with the equity/crypto panels but is *not* a tradable session boundary;
|
||
# broker-specific 5pm-NY cutoffs are wired in `case_studies/fx_pairs/`
|
||
# downstream.
|
||
#
|
||
# **Next**: `13_data_quality_framework` profiles the cross-asset DQ checks
|
||
# that consume this panel and the others built up so far.
|