wehub-resource-sync
855 lines
37 KiB
Plaintext
855 lines
37 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "0e6cf5e5",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.001987,
|
||
"end_time": "2026-06-13T02:33:42.350184+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:33:42.348197+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"# FX Pairs — Exploratory Data Analysis\n",
|
||
"\n",
|
||
"**Docker image**: `ml4t`\n",
|
||
"\n",
|
||
"## Purpose\n",
|
||
"Profile the OANDA 20-pair, 4-hour FX dataset that anchors the FX case study.\n",
|
||
"FX is OTC: there is no central tape, so quotes and reported volumes are\n",
|
||
"venue-specific. The notebook surveys coverage, quote conventions, OHLC\n",
|
||
"integrity, and the 4h→daily aggregation used by downstream chapters.\n",
|
||
"\n",
|
||
"## Learning Objectives\n",
|
||
"- Load and inspect the 4-hour OHLC + indicative-volume panel for 20 pairs.\n",
|
||
"- Distinguish direct (USD-quoted), indirect (USD-base), and cross pairs.\n",
|
||
"- Read FX volume as an OANDA indicator, not an authoritative tape.\n",
|
||
"- Aggregate 4h bars to UTC-day daily bars and read the gap-pattern signal.\n",
|
||
"\n",
|
||
"## Book reference\n",
|
||
"Chapter 2, §2.2 (asset-class market data — foreign exchange). The FX case\n",
|
||
"study built on this dataset lives in `case_studies/fx_pairs/`.\n",
|
||
"\n",
|
||
"## Prerequisites\n",
|
||
"- OANDA 4h FX parquet files materialized under `ML4T_DATA_PATH`.\n",
|
||
"- Loader `data.load_fx_pairs`."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 1,
|
||
"id": "75a4c694",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:33:42.353826Z",
|
||
"iopub.status.busy": "2026-06-13T02:33:42.353718Z",
|
||
"iopub.status.idle": "2026-06-13T02:33:43.581552Z",
|
||
"shell.execute_reply": "2026-06-13T02:33:43.581002Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 1.230464,
|
||
"end_time": "2026-06-13T02:33:43.582246+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:33:42.351782+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"\"\"\"FX Pairs — Exploratory data analysis of OANDA currency pair data.\"\"\"\n",
|
||
"\n",
|
||
"import polars as pl\n",
|
||
"\n",
|
||
"from data import load_fx_pairs\n",
|
||
"from utils.data_quality import check_ohlc_invariants, per_asset_stats"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 2,
|
||
"id": "26b9a4ca",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:33:43.586308Z",
|
||
"iopub.status.busy": "2026-06-13T02:33:43.586220Z",
|
||
"iopub.status.idle": "2026-06-13T02:33:43.588185Z",
|
||
"shell.execute_reply": "2026-06-13T02:33:43.587686Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.004652,
|
||
"end_time": "2026-06-13T02:33:43.588663+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:33:43.584011+00:00",
|
||
"status": "completed"
|
||
},
|
||
"tags": [
|
||
"parameters"
|
||
]
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Production defaults — Papermill injects overrides for CI\n",
|
||
"# (No tunable knobs: this notebook EDAs the full 12-pair universe via the\n",
|
||
"# canonical load_fx_pairs() API; there is no MAX_SYMBOLS / START_DATE knob to expose.)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "d3ace9f3",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.000828,
|
||
"end_time": "2026-06-13T02:33:43.590412+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:33:43.589584+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"## 1. Load and Inspect"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 3,
|
||
"id": "e092607c",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:33:43.592603Z",
|
||
"iopub.status.busy": "2026-06-13T02:33:43.592534Z",
|
||
"iopub.status.idle": "2026-06-13T02:33:43.645036Z",
|
||
"shell.execute_reply": "2026-06-13T02:33:43.644439Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.054124,
|
||
"end_time": "2026-06-13T02:33:43.645352+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:33:43.591228+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"=== FX Dataset ===\n",
|
||
"Shape: (478640, 7)\n",
|
||
"Columns: ['timestamp', 'symbol', 'open', 'high', 'low', 'close', 'volume']\n",
|
||
"Date range: 2011-01-02 14:00:00 to 2025-12-31 18:00:00\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"fx_4h = load_fx_pairs(frequency=\"4h\")\n",
|
||
"\n",
|
||
"print(\"=== FX Dataset ===\")\n",
|
||
"print(f\"Shape: {fx_4h.shape}\")\n",
|
||
"print(f\"Columns: {fx_4h.columns}\")\n",
|
||
"print(f\"Date range: {fx_4h['timestamp'].min()} to {fx_4h['timestamp'].max()}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "226404df",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.000883,
|
||
"end_time": "2026-06-13T02:33:43.647282+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:33:43.646399+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"### Volume Disclaimer\n",
|
||
"\n",
|
||
"**Important**: FX is an OTC market. Volume figures are indicative estimates from OANDA,\n",
|
||
"not authoritative exchange data. Do not interpret FX volume the same way as equity volume."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 4,
|
||
"id": "582f10dd",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:33:43.649708Z",
|
||
"iopub.status.busy": "2026-06-13T02:33:43.649581Z",
|
||
"iopub.status.idle": "2026-06-13T02:33:43.653412Z",
|
||
"shell.execute_reply": "2026-06-13T02:33:43.652994Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.00548,
|
||
"end_time": "2026-06-13T02:33:43.653645+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:33:43.648165+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/html": [
|
||
"<div><style>\n",
|
||
".dataframe > thead > tr,\n",
|
||
".dataframe > tbody > tr {\n",
|
||
" text-align: right;\n",
|
||
" white-space: pre-wrap;\n",
|
||
"}\n",
|
||
"</style>\n",
|
||
"<small>shape: (5, 7)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>timestamp</th><th>symbol</th><th>open</th><th>high</th><th>low</th><th>close</th><th>volume</th></tr><tr><td>datetime[ms]</td><td>str</td><td>f64</td><td>f64</td><td>f64</td><td>f64</td><td>i64</td></tr></thead><tbody><tr><td>2011-01-02 18:00:00</td><td>"AUD_JPY"</td><td>83.0</td><td>83.332</td><td>82.8</td><td>82.875</td><td>986</td></tr><tr><td>2011-01-02 22:00:00</td><td>"AUD_JPY"</td><td>82.877</td><td>83.015</td><td>82.698</td><td>82.796</td><td>4378</td></tr><tr><td>2011-01-03 02:00:00</td><td>"AUD_JPY"</td><td>82.8</td><td>82.98</td><td>82.782</td><td>82.924</td><td>1934</td></tr><tr><td>2011-01-03 06:00:00</td><td>"AUD_JPY"</td><td>82.926</td><td>83.03</td><td>82.826</td><td>83.03</td><td>3949</td></tr><tr><td>2011-01-03 10:00:00</td><td>"AUD_JPY"</td><td>83.032</td><td>83.158</td><td>82.986</td><td>83.01</td><td>5296</td></tr></tbody></table></div>"
|
||
],
|
||
"text/plain": [
|
||
"shape: (5, 7)\n",
|
||
"┌─────────────────────┬─────────┬────────┬────────┬────────┬────────┬────────┐\n",
|
||
"│ timestamp ┆ symbol ┆ open ┆ high ┆ low ┆ close ┆ volume │\n",
|
||
"│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │\n",
|
||
"│ datetime[ms] ┆ str ┆ f64 ┆ f64 ┆ f64 ┆ f64 ┆ i64 │\n",
|
||
"╞═════════════════════╪═════════╪════════╪════════╪════════╪════════╪════════╡\n",
|
||
"│ 2011-01-02 18:00:00 ┆ AUD_JPY ┆ 83.0 ┆ 83.332 ┆ 82.8 ┆ 82.875 ┆ 986 │\n",
|
||
"│ 2011-01-02 22:00:00 ┆ AUD_JPY ┆ 82.877 ┆ 83.015 ┆ 82.698 ┆ 82.796 ┆ 4378 │\n",
|
||
"│ 2011-01-03 02:00:00 ┆ AUD_JPY ┆ 82.8 ┆ 82.98 ┆ 82.782 ┆ 82.924 ┆ 1934 │\n",
|
||
"│ 2011-01-03 06:00:00 ┆ AUD_JPY ┆ 82.926 ┆ 83.03 ┆ 82.826 ┆ 83.03 ┆ 3949 │\n",
|
||
"│ 2011-01-03 10:00:00 ┆ AUD_JPY ┆ 83.032 ┆ 83.158 ┆ 82.986 ┆ 83.01 ┆ 5296 │\n",
|
||
"└─────────────────────┴─────────┴────────┴────────┴────────┴────────┴────────┘"
|
||
]
|
||
},
|
||
"execution_count": 4,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"fx_4h.head()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 5,
|
||
"id": "97391cf2",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:33:43.656035Z",
|
||
"iopub.status.busy": "2026-06-13T02:33:43.655974Z",
|
||
"iopub.status.idle": "2026-06-13T02:33:43.661039Z",
|
||
"shell.execute_reply": "2026-06-13T02:33:43.660623Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.006656,
|
||
"end_time": "2026-06-13T02:33:43.661267+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:33:43.654611+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"\n",
|
||
"Currency pairs (20):\n",
|
||
" AUD_JPY\n",
|
||
" AUD_NZD\n",
|
||
" AUD_USD\n",
|
||
" CAD_JPY\n",
|
||
" CHF_JPY\n",
|
||
" EUR_AUD\n",
|
||
" EUR_CAD\n",
|
||
" EUR_CHF\n",
|
||
" EUR_GBP\n",
|
||
" EUR_JPY\n",
|
||
" EUR_USD\n",
|
||
" GBP_AUD\n",
|
||
" GBP_CHF\n",
|
||
" GBP_JPY\n",
|
||
" GBP_USD\n",
|
||
" NZD_JPY\n",
|
||
" NZD_USD\n",
|
||
" USD_CAD\n",
|
||
" USD_CHF\n",
|
||
" USD_JPY\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Available pairs\n",
|
||
"pairs = fx_4h[\"symbol\"].unique().sort().to_list()\n",
|
||
"print(f\"\\nCurrency pairs ({len(pairs)}):\")\n",
|
||
"for pair in pairs:\n",
|
||
" print(f\" {pair}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "95116a76",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.000895,
|
||
"end_time": "2026-06-13T02:33:43.663133+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:33:43.662238+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"### Symbol Normalization\n",
|
||
"\n",
|
||
"The data uses underscore format (`EUR_USD`). The canonical format for this dataset is\n",
|
||
"concatenated (`EURUSD`). Here's how to convert:"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 6,
|
||
"id": "aea13dc3",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:33:43.665439Z",
|
||
"iopub.status.busy": "2026-06-13T02:33:43.665377Z",
|
||
"iopub.status.idle": "2026-06-13T02:33:43.677943Z",
|
||
"shell.execute_reply": "2026-06-13T02:33:43.677539Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.014182,
|
||
"end_time": "2026-06-13T02:33:43.678196+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:33:43.664014+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Symbol normalization example:\n",
|
||
" Raw format: EUR_USD, USD_JPY, GBP_USD\n",
|
||
" Canonical: EURUSD, USDJPY, GBPUSD\n",
|
||
"\n",
|
||
"Normalized pairs: ['AUDJPY', 'AUDNZD', 'AUDUSD', 'CADJPY', 'CHFJPY', 'EURAUD'] ...\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Normalize symbols: EUR_USD → EURUSD (canonical format)\n",
|
||
"# The raw file uses underscores; we normalize to concatenated format for downstream joins\n",
|
||
"fx = fx_4h.with_columns(pl.col(\"symbol\").str.replace(\"_\", \"\").alias(\"symbol\"))\n",
|
||
"\n",
|
||
"print(\"Symbol normalization example:\")\n",
|
||
"print(\" Raw format: EUR_USD, USD_JPY, GBP_USD\")\n",
|
||
"print(\" Canonical: EURUSD, USDJPY, GBPUSD\")\n",
|
||
"print(f\"\\nNormalized pairs: {fx['symbol'].unique().sort().to_list()[:6]} ...\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "a1437f5f",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.000954,
|
||
"end_time": "2026-06-13T02:33:43.680241+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:33:43.679287+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"## 2. Coverage Summary"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 7,
|
||
"id": "bb1c9769",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:33:43.682634Z",
|
||
"iopub.status.busy": "2026-06-13T02:33:43.682558Z",
|
||
"iopub.status.idle": "2026-06-13T02:33:43.693125Z",
|
||
"shell.execute_reply": "2026-06-13T02:33:43.692663Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.012245,
|
||
"end_time": "2026-06-13T02:33:43.693424+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:33:43.681179+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/html": [
|
||
"<div><style>\n",
|
||
".dataframe > thead > tr,\n",
|
||
".dataframe > tbody > tr {\n",
|
||
" text-align: right;\n",
|
||
" white-space: pre-wrap;\n",
|
||
"}\n",
|
||
"</style>\n",
|
||
"<small>shape: (20, 6)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>symbol</th><th>rows</th><th>start</th><th>end</th><th>avg_price</th><th>avg_volume</th></tr><tr><td>str</td><td>u32</td><td>datetime[ms]</td><td>datetime[ms]</td><td>f64</td><td>f64</td></tr></thead><tbody><tr><td>"GBPAUD"</td><td>23945</td><td>2011-01-02 14:00:00</td><td>2025-12-31 18:00:00</td><td>1.795271</td><td>27719.096262</td></tr><tr><td>"GBPJPY"</td><td>23924</td><td>2011-01-02 18:00:00</td><td>2025-12-31 18:00:00</td><td>156.656959</td><td>22954.817673</td></tr><tr><td>"EURCAD"</td><td>23921</td><td>2011-01-02 14:00:00</td><td>2025-12-31 18:00:00</td><td>1.44887</td><td>21140.860081</td></tr><tr><td>"GBPCHF"</td><td>23924</td><td>2011-01-02 18:00:00</td><td>2025-12-31 18:00:00</td><td>1.302496</td><td>18534.499415</td></tr><tr><td>"CHFJPY"</td><td>23919</td><td>2011-01-02 18:00:00</td><td>2025-12-31 18:00:00</td><td>122.492732</td><td>18296.437686</td></tr><tr><td>…</td><td>…</td><td>…</td><td>…</td><td>…</td><td>…</td></tr><tr><td>"NZDUSD"</td><td>23943</td><td>2011-01-02 14:00:00</td><td>2025-12-31 18:00:00</td><td>0.702921</td><td>7312.247797</td></tr><tr><td>"EURCHF"</td><td>23925</td><td>2011-01-02 14:00:00</td><td>2025-12-31 18:00:00</td><td>1.099216</td><td>6110.44698</td></tr><tr><td>"EURGBP"</td><td>23922</td><td>2011-01-02 14:00:00</td><td>2025-12-31 18:00:00</td><td>0.846666</td><td>5989.497952</td></tr><tr><td>"AUDUSD"</td><td>23941</td><td>2011-01-02 18:00:00</td><td>2025-12-31 18:00:00</td><td>0.789923</td><td>5563.880832</td></tr><tr><td>"USDCHF"</td><td>23927</td><td>2011-01-02 14:00:00</td><td>2025-12-31 18:00:00</td><td>0.93156</td><td>5375.534835</td></tr></tbody></table></div>"
|
||
],
|
||
"text/plain": [
|
||
"shape: (20, 6)\n",
|
||
"┌────────┬───────┬─────────────────────┬─────────────────────┬────────────┬──────────────┐\n",
|
||
"│ symbol ┆ rows ┆ start ┆ end ┆ avg_price ┆ avg_volume │\n",
|
||
"│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │\n",
|
||
"│ str ┆ u32 ┆ datetime[ms] ┆ datetime[ms] ┆ f64 ┆ f64 │\n",
|
||
"╞════════╪═══════╪═════════════════════╪═════════════════════╪════════════╪══════════════╡\n",
|
||
"│ GBPAUD ┆ 23945 ┆ 2011-01-02 14:00:00 ┆ 2025-12-31 18:00:00 ┆ 1.795271 ┆ 27719.096262 │\n",
|
||
"│ GBPJPY ┆ 23924 ┆ 2011-01-02 18:00:00 ┆ 2025-12-31 18:00:00 ┆ 156.656959 ┆ 22954.817673 │\n",
|
||
"│ EURCAD ┆ 23921 ┆ 2011-01-02 14:00:00 ┆ 2025-12-31 18:00:00 ┆ 1.44887 ┆ 21140.860081 │\n",
|
||
"│ GBPCHF ┆ 23924 ┆ 2011-01-02 18:00:00 ┆ 2025-12-31 18:00:00 ┆ 1.302496 ┆ 18534.499415 │\n",
|
||
"│ CHFJPY ┆ 23919 ┆ 2011-01-02 18:00:00 ┆ 2025-12-31 18:00:00 ┆ 122.492732 ┆ 18296.437686 │\n",
|
||
"│ … ┆ … ┆ … ┆ … ┆ … ┆ … │\n",
|
||
"│ NZDUSD ┆ 23943 ┆ 2011-01-02 14:00:00 ┆ 2025-12-31 18:00:00 ┆ 0.702921 ┆ 7312.247797 │\n",
|
||
"│ EURCHF ┆ 23925 ┆ 2011-01-02 14:00:00 ┆ 2025-12-31 18:00:00 ┆ 1.099216 ┆ 6110.44698 │\n",
|
||
"│ EURGBP ┆ 23922 ┆ 2011-01-02 14:00:00 ┆ 2025-12-31 18:00:00 ┆ 0.846666 ┆ 5989.497952 │\n",
|
||
"│ AUDUSD ┆ 23941 ┆ 2011-01-02 18:00:00 ┆ 2025-12-31 18:00:00 ┆ 0.789923 ┆ 5563.880832 │\n",
|
||
"│ USDCHF ┆ 23927 ┆ 2011-01-02 14:00:00 ┆ 2025-12-31 18:00:00 ┆ 0.93156 ┆ 5375.534835 │\n",
|
||
"└────────┴───────┴─────────────────────┴─────────────────────┴────────────┴──────────────┘"
|
||
]
|
||
},
|
||
"execution_count": 7,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"# Per-pair statistics (using normalized symbols)\n",
|
||
"pairs = fx[\"symbol\"].unique().sort().to_list()\n",
|
||
"\n",
|
||
"pair_stats = per_asset_stats(\n",
|
||
" fx,\n",
|
||
" time_col=\"timestamp\",\n",
|
||
" asset_col=\"symbol\",\n",
|
||
" price_col=\"close\",\n",
|
||
" volume_col=\"volume\",\n",
|
||
")\n",
|
||
"\n",
|
||
"pair_stats.sort(\"avg_volume\", descending=True)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "736a507b",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.001009,
|
||
"end_time": "2026-06-13T02:33:43.695620+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:33:43.694611+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"## 3. Quote Conventions\n",
|
||
"\n",
|
||
"FX pairs follow **BASE/QUOTE** convention:\n",
|
||
"\n",
|
||
"| Pair | Interpretation | USD Strength |\n",
|
||
"|------|----------------|--------------|\n",
|
||
"| EUR/USD = 1.10 | 1 EUR costs 1.10 USD | Down = USD stronger |\n",
|
||
"| USD/JPY = 150 | 1 USD costs 150 JPY | Up = USD stronger |\n",
|
||
"| EUR/GBP = 0.86 | 1 EUR costs 0.86 GBP | Cross rate (no USD) |\n",
|
||
"\n",
|
||
"To create a consistent \"USD index\", you must **invert** EUR/USD and GBP/USD."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 8,
|
||
"id": "1248d371",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:33:43.698207Z",
|
||
"iopub.status.busy": "2026-06-13T02:33:43.698121Z",
|
||
"iopub.status.idle": "2026-06-13T02:33:43.701090Z",
|
||
"shell.execute_reply": "2026-06-13T02:33:43.700792Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.004731,
|
||
"end_time": "2026-06-13T02:33:43.701341+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:33:43.696610+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/html": [
|
||
"<div><style>\n",
|
||
".dataframe > thead > tr,\n",
|
||
".dataframe > tbody > tr {\n",
|
||
" text-align: right;\n",
|
||
" white-space: pre-wrap;\n",
|
||
"}\n",
|
||
"</style>\n",
|
||
"<small>shape: (9, 4)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>symbol</th><th>convention</th><th>meaning</th><th>usd_strength</th></tr><tr><td>str</td><td>str</td><td>str</td><td>str</td></tr></thead><tbody><tr><td>"EURUSD"</td><td>"Direct"</td><td>"USD per EUR"</td><td>"invert for USD strength"</td></tr><tr><td>"GBPUSD"</td><td>"Direct"</td><td>"USD per GBP"</td><td>"invert for USD strength"</td></tr><tr><td>"AUDUSD"</td><td>"Direct"</td><td>"USD per AUD"</td><td>"invert for USD strength"</td></tr><tr><td>"NZDUSD"</td><td>"Direct"</td><td>"USD per NZD"</td><td>"invert for USD strength"</td></tr><tr><td>"USDJPY"</td><td>"Indirect"</td><td>"JPY per USD"</td><td>"direct USD strength"</td></tr><tr><td>"USDCHF"</td><td>"Indirect"</td><td>"CHF per USD"</td><td>"direct USD strength"</td></tr><tr><td>"USDCAD"</td><td>"Indirect"</td><td>"CAD per USD"</td><td>"direct USD strength"</td></tr><tr><td>"EURGBP"</td><td>"Cross"</td><td>"GBP per EUR"</td><td>"no USD"</td></tr><tr><td>"EURJPY"</td><td>"Cross"</td><td>"JPY per EUR"</td><td>"no USD"</td></tr></tbody></table></div>"
|
||
],
|
||
"text/plain": [
|
||
"shape: (9, 4)\n",
|
||
"┌────────┬────────────┬─────────────┬─────────────────────────┐\n",
|
||
"│ symbol ┆ convention ┆ meaning ┆ usd_strength │\n",
|
||
"│ --- ┆ --- ┆ --- ┆ --- │\n",
|
||
"│ str ┆ str ┆ str ┆ str │\n",
|
||
"╞════════╪════════════╪═════════════╪═════════════════════════╡\n",
|
||
"│ EURUSD ┆ Direct ┆ USD per EUR ┆ invert for USD strength │\n",
|
||
"│ GBPUSD ┆ Direct ┆ USD per GBP ┆ invert for USD strength │\n",
|
||
"│ AUDUSD ┆ Direct ┆ USD per AUD ┆ invert for USD strength │\n",
|
||
"│ NZDUSD ┆ Direct ┆ USD per NZD ┆ invert for USD strength │\n",
|
||
"│ USDJPY ┆ Indirect ┆ JPY per USD ┆ direct USD strength │\n",
|
||
"│ USDCHF ┆ Indirect ┆ CHF per USD ┆ direct USD strength │\n",
|
||
"│ USDCAD ┆ Indirect ┆ CAD per USD ┆ direct USD strength │\n",
|
||
"│ EURGBP ┆ Cross ┆ GBP per EUR ┆ no USD │\n",
|
||
"│ EURJPY ┆ Cross ┆ JPY per EUR ┆ no USD │\n",
|
||
"└────────┴────────────┴─────────────┴─────────────────────────┘"
|
||
]
|
||
},
|
||
"execution_count": 8,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"# Quote convention classification (using canonical symbols).\n",
|
||
"QUOTE_CONVENTIONS = {\n",
|
||
" \"EURUSD\": (\"Direct\", \"USD per EUR\", \"invert for USD strength\"),\n",
|
||
" \"GBPUSD\": (\"Direct\", \"USD per GBP\", \"invert for USD strength\"),\n",
|
||
" \"AUDUSD\": (\"Direct\", \"USD per AUD\", \"invert for USD strength\"),\n",
|
||
" \"NZDUSD\": (\"Direct\", \"USD per NZD\", \"invert for USD strength\"),\n",
|
||
" \"USDJPY\": (\"Indirect\", \"JPY per USD\", \"direct USD strength\"),\n",
|
||
" \"USDCHF\": (\"Indirect\", \"CHF per USD\", \"direct USD strength\"),\n",
|
||
" \"USDCAD\": (\"Indirect\", \"CAD per USD\", \"direct USD strength\"),\n",
|
||
" \"EURGBP\": (\"Cross\", \"GBP per EUR\", \"no USD\"),\n",
|
||
" \"EURJPY\": (\"Cross\", \"JPY per EUR\", \"no USD\"),\n",
|
||
"}\n",
|
||
"\n",
|
||
"quote_conventions = pl.DataFrame(\n",
|
||
" [\n",
|
||
" {\"symbol\": p, \"convention\": c, \"meaning\": m, \"usd_strength\": n}\n",
|
||
" for p, (c, m, n) in QUOTE_CONVENTIONS.items()\n",
|
||
" if p in pairs\n",
|
||
" ]\n",
|
||
")\n",
|
||
"quote_conventions"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "f6e41bbb",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.0011,
|
||
"end_time": "2026-06-13T02:33:43.703547+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:33:43.702447+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"## 4. Data Quality"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 9,
|
||
"id": "4476f44a",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:33:43.706165Z",
|
||
"iopub.status.busy": "2026-06-13T02:33:43.706091Z",
|
||
"iopub.status.idle": "2026-06-13T02:33:43.713408Z",
|
||
"shell.execute_reply": "2026-06-13T02:33:43.712933Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.009113,
|
||
"end_time": "2026-06-13T02:33:43.713719+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:33:43.704606+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/html": [
|
||
"<div><style>\n",
|
||
".dataframe > thead > tr,\n",
|
||
".dataframe > tbody > tr {\n",
|
||
" text-align: right;\n",
|
||
" white-space: pre-wrap;\n",
|
||
"}\n",
|
||
"</style>\n",
|
||
"<small>shape: (6, 4)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>check</th><th>valid_pct</th><th>applicable_rows</th><th>total_rows</th></tr><tr><td>str</td><td>f64</td><td>i64</td><td>i64</td></tr></thead><tbody><tr><td>"high_gte_low"</td><td>100.0</td><td>478640</td><td>478640</td></tr><tr><td>"high_gte_open"</td><td>100.0</td><td>478640</td><td>478640</td></tr><tr><td>"high_gte_close"</td><td>100.0</td><td>478640</td><td>478640</td></tr><tr><td>"low_lte_open"</td><td>100.0</td><td>478640</td><td>478640</td></tr><tr><td>"low_lte_close"</td><td>100.0</td><td>478640</td><td>478640</td></tr><tr><td>"volume_non_negative"</td><td>100.0</td><td>478640</td><td>478640</td></tr></tbody></table></div>"
|
||
],
|
||
"text/plain": [
|
||
"shape: (6, 4)\n",
|
||
"┌─────────────────────┬───────────┬─────────────────┬────────────┐\n",
|
||
"│ check ┆ valid_pct ┆ applicable_rows ┆ total_rows │\n",
|
||
"│ --- ┆ --- ┆ --- ┆ --- │\n",
|
||
"│ str ┆ f64 ┆ i64 ┆ i64 │\n",
|
||
"╞═════════════════════╪═══════════╪═════════════════╪════════════╡\n",
|
||
"│ high_gte_low ┆ 100.0 ┆ 478640 ┆ 478640 │\n",
|
||
"│ high_gte_open ┆ 100.0 ┆ 478640 ┆ 478640 │\n",
|
||
"│ high_gte_close ┆ 100.0 ┆ 478640 ┆ 478640 │\n",
|
||
"│ low_lte_open ┆ 100.0 ┆ 478640 ┆ 478640 │\n",
|
||
"│ low_lte_close ┆ 100.0 ┆ 478640 ┆ 478640 │\n",
|
||
"│ volume_non_negative ┆ 100.0 ┆ 478640 ┆ 478640 │\n",
|
||
"└─────────────────────┴───────────┴─────────────────┴────────────┘"
|
||
]
|
||
},
|
||
"execution_count": 9,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"# OHLC invariants\n",
|
||
"invariants = check_ohlc_invariants(fx)\n",
|
||
"invariants"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 10,
|
||
"id": "9c27cc3c",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:33:43.717828Z",
|
||
"iopub.status.busy": "2026-06-13T02:33:43.717708Z",
|
||
"iopub.status.idle": "2026-06-13T02:33:43.724040Z",
|
||
"shell.execute_reply": "2026-06-13T02:33:43.723475Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.009097,
|
||
"end_time": "2026-06-13T02:33:43.724412+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:33:43.715315+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"\n",
|
||
"Gaps > 24 hours (EURUSD): 747 (should be weekends only)\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Check for weekend gaps (expected in FX, which trades 24/5)\n",
|
||
"eurusd = fx.filter(pl.col(\"symbol\") == \"EURUSD\").sort(\"timestamp\")\n",
|
||
"\n",
|
||
"eurusd_gaps = eurusd.with_columns(\n",
|
||
" pl.col(\"timestamp\").diff().dt.total_hours().alias(\"hours_since_prev\")\n",
|
||
")\n",
|
||
"\n",
|
||
"large_gaps = eurusd_gaps.filter(pl.col(\"hours_since_prev\") > 24)\n",
|
||
"print(f\"\\nGaps > 24 hours (EURUSD): {len(large_gaps)} (should be weekends only)\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "b6e1c1bb",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.001464,
|
||
"end_time": "2026-06-13T02:33:43.727484+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:33:43.726020+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"## 5. Daily Aggregation\n",
|
||
"\n",
|
||
"Aggregate 4-hour bars to daily for consistency with other datasets.\n",
|
||
"\n",
|
||
"**Note**: This uses UTC midnight boundaries. FX daily bars are conventionally defined\n",
|
||
"by a session cutoff (often 5pm New York). For production, align to your broker's\n",
|
||
"convention. This simple calendar-day aggregation is sufficient for exploration."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 11,
|
||
"id": "1c719ab9",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:33:43.730960Z",
|
||
"iopub.status.busy": "2026-06-13T02:33:43.730762Z",
|
||
"iopub.status.idle": "2026-06-13T02:33:43.773847Z",
|
||
"shell.execute_reply": "2026-06-13T02:33:43.773341Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.045315,
|
||
"end_time": "2026-06-13T02:33:43.774176+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:33:43.728861+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Daily aggregation (UTC boundaries) — shape: (94642, 7)\n"
|
||
]
|
||
},
|
||
{
|
||
"data": {
|
||
"text/html": [
|
||
"<div><style>\n",
|
||
".dataframe > thead > tr,\n",
|
||
".dataframe > tbody > tr {\n",
|
||
" text-align: right;\n",
|
||
" white-space: pre-wrap;\n",
|
||
"}\n",
|
||
"</style>\n",
|
||
"<small>shape: (5, 7)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>symbol</th><th>timestamp</th><th>open</th><th>high</th><th>low</th><th>close</th><th>volume</th></tr><tr><td>str</td><td>datetime[ms]</td><td>f64</td><td>f64</td><td>f64</td><td>f64</td><td>i64</td></tr></thead><tbody><tr><td>"EURUSD"</td><td>2025-12-26 00:00:00</td><td>1.17873</td><td>1.17968</td><td>1.17616</td><td>1.1771</td><td>99014</td></tr><tr><td>"EURUSD"</td><td>2025-12-28 00:00:00</td><td>1.1768</td><td>1.17864</td><td>1.17654</td><td>1.17842</td><td>10811</td></tr><tr><td>"EURUSD"</td><td>2025-12-29 00:00:00</td><td>1.1784</td><td>1.17891</td><td>1.17494</td><td>1.17751</td><td>110397</td></tr><tr><td>"EURUSD"</td><td>2025-12-30 00:00:00</td><td>1.1775</td><td>1.17798</td><td>1.17419</td><td>1.17426</td><td>83060</td></tr><tr><td>"EURUSD"</td><td>2025-12-31 00:00:00</td><td>1.17424</td><td>1.17595</td><td>1.172</td><td>1.17459</td><td>92505</td></tr></tbody></table></div>"
|
||
],
|
||
"text/plain": [
|
||
"shape: (5, 7)\n",
|
||
"┌────────┬─────────────────────┬─────────┬─────────┬─────────┬─────────┬────────┐\n",
|
||
"│ symbol ┆ timestamp ┆ open ┆ high ┆ low ┆ close ┆ volume │\n",
|
||
"│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │\n",
|
||
"│ str ┆ datetime[ms] ┆ f64 ┆ f64 ┆ f64 ┆ f64 ┆ i64 │\n",
|
||
"╞════════╪═════════════════════╪═════════╪═════════╪═════════╪═════════╪════════╡\n",
|
||
"│ EURUSD ┆ 2025-12-26 00:00:00 ┆ 1.17873 ┆ 1.17968 ┆ 1.17616 ┆ 1.1771 ┆ 99014 │\n",
|
||
"│ EURUSD ┆ 2025-12-28 00:00:00 ┆ 1.1768 ┆ 1.17864 ┆ 1.17654 ┆ 1.17842 ┆ 10811 │\n",
|
||
"│ EURUSD ┆ 2025-12-29 00:00:00 ┆ 1.1784 ┆ 1.17891 ┆ 1.17494 ┆ 1.17751 ┆ 110397 │\n",
|
||
"│ EURUSD ┆ 2025-12-30 00:00:00 ┆ 1.1775 ┆ 1.17798 ┆ 1.17419 ┆ 1.17426 ┆ 83060 │\n",
|
||
"│ EURUSD ┆ 2025-12-31 00:00:00 ┆ 1.17424 ┆ 1.17595 ┆ 1.172 ┆ 1.17459 ┆ 92505 │\n",
|
||
"└────────┴─────────────────────┴─────────┴─────────┴─────────┴─────────┴────────┘"
|
||
]
|
||
},
|
||
"execution_count": 11,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"# Daily aggregation (must be sorted for group_by_dynamic)\n",
|
||
"fx_daily = (\n",
|
||
" fx.sort(\"symbol\", \"timestamp\")\n",
|
||
" .group_by_dynamic(\"timestamp\", every=\"1d\", group_by=\"symbol\")\n",
|
||
" .agg(\n",
|
||
" [\n",
|
||
" pl.col(\"open\").first(),\n",
|
||
" pl.col(\"high\").max(),\n",
|
||
" pl.col(\"low\").min(),\n",
|
||
" pl.col(\"close\").last(),\n",
|
||
" pl.col(\"volume\").sum(),\n",
|
||
" ]\n",
|
||
" )\n",
|
||
")\n",
|
||
"\n",
|
||
"print(f\"Daily aggregation (UTC boundaries) — shape: {fx_daily.shape}\")\n",
|
||
"fx_daily.filter(pl.col(\"symbol\") == \"EURUSD\").tail(5)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "28013e00",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.001196,
|
||
"end_time": "2026-06-13T02:33:43.776700+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:33:43.775504+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"## Key Takeaways\n",
|
||
"\n",
|
||
"Profile of the OANDA 4h FX panel that anchors the FX case study.\n",
|
||
"\n",
|
||
"### Quantitative Findings\n",
|
||
"- **Panel scale**: 478,640 4h observations across 20 currency pairs spanning\n",
|
||
" 2011-01-02 → 2025-12-31. Each pair has ~23,920–23,945 4h bars.\n",
|
||
"- **Liquidity tiers (by indicative OANDA volume)**: GBPAUD, GBPJPY, EURCAD,\n",
|
||
" GBPCHF and CHFJPY top the table at 18k–28k contracts/4h; the bottom of\n",
|
||
" the universe (NZDUSD, EURCHF, EURGBP, AUDUSD, USDCHF) sits at 5k–7k.\n",
|
||
" These rankings are *OANDA-specific* — interbank-market liquidity for\n",
|
||
" EURUSD/USDJPY is the largest globally but is not visible to a single\n",
|
||
" retail venue.\n",
|
||
"- **OHLC integrity**: 100% of 4h bars satisfy all six invariants\n",
|
||
" (high ≥ low/open/close, low ≤ open/close, volume ≥ 0).\n",
|
||
"- **Session gaps**: 747 EURUSD inter-bar gaps exceed 24 h, matching the\n",
|
||
" ~770 weekend closes over 14 years — confirming the 24/5 calendar.\n",
|
||
"- **Daily roll-up**: UTC-boundary aggregation produces 94,642 daily rows\n",
|
||
" across the panel (≈4,732 trading days × 20 pairs).\n",
|
||
"\n",
|
||
"### Implications for Practitioners\n",
|
||
"- **Volume**: Treat as a relative liquidity indicator across pairs on this\n",
|
||
" venue, not as an interbank tape.\n",
|
||
"- **Quote inversion**: USD strength composites must invert direct pairs\n",
|
||
" (EUR/USD, GBP/USD, AUD/USD, NZD/USD); USD-base and cross pairs do not\n",
|
||
" need inversion.\n",
|
||
"- **Daily session convention**: UTC-day aggregation is convenient for joins\n",
|
||
" with the equity/crypto panels but is *not* a tradable session boundary;\n",
|
||
" broker-specific 5pm-NY cutoffs are wired in `case_studies/fx_pairs/`\n",
|
||
" downstream.\n",
|
||
"\n",
|
||
"**Next**: `13_data_quality_framework` profiles the cross-asset DQ checks\n",
|
||
"that consume this panel and the others built up so far."
|
||
]
|
||
}
|
||
],
|
||
"metadata": {
|
||
"jupytext": {
|
||
"cell_metadata_filter": "tags,-all"
|
||
},
|
||
"kernelspec": {
|
||
"display_name": "Python 3 (ipykernel)",
|
||
"language": "python",
|
||
"name": "python3"
|
||
},
|
||
"language_info": {
|
||
"codemirror_mode": {
|
||
"name": "ipython",
|
||
"version": 3
|
||
},
|
||
"file_extension": ".py",
|
||
"mimetype": "text/x-python",
|
||
"name": "python",
|
||
"nbconvert_exporter": "python",
|
||
"pygments_lexer": "ipython3",
|
||
"version": "3.14.3"
|
||
},
|
||
"papermill": {
|
||
"default_parameters": {},
|
||
"duration": 2.888138,
|
||
"end_time": "2026-06-13T02:33:44.593918+00:00",
|
||
"environment_variables": {},
|
||
"exception": null,
|
||
"input_path": "02_financial_data_universe/12_fx_pairs_eda.ipynb",
|
||
"output_path": "02_financial_data_universe/12_fx_pairs_eda.ipynb",
|
||
"parameters": {},
|
||
"start_time": "2026-06-13T02:33:41.705780+00:00",
|
||
"version": "2.7.0"
|
||
},
|
||
"ml4t_provenance": {
|
||
"source_py_blob": "8f3afbd31633079f8ac78d86f3bc5a9cf4b326ad",
|
||
"executed_at": "2026-06-13T02:33:44.819485+00:00",
|
||
"executor": "cpu-local",
|
||
"production": true,
|
||
"parameters": {},
|
||
"notes": "executed-state finalization batch (2026-06-13 run)"
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 5
|
||
}
|