wehub-resource-sync
848 lines
27 KiB
Plaintext
848 lines
27 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "34eda0f3",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.002064,
|
||
"end_time": "2026-06-13T02:33:28.977337+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:33:28.975273+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"# Crypto Perps — Exploratory Data Analysis\n",
|
||
"\n",
|
||
"**Docker image**: `ml4t`\n",
|
||
"\n",
|
||
"## Purpose\n",
|
||
"\n",
|
||
"Profile the Binance Futures hourly OHLCV dataset for 19 perpetual contracts\n",
|
||
"alongside the 8-hourly Premium Index that captures the perpetual–spot\n",
|
||
"basis. The notebook anchors data shape, units, coverage, and OHLC integrity\n",
|
||
"before the strategy work begins in Chapter 6.\n",
|
||
"\n",
|
||
"## Learning Objectives\n",
|
||
"\n",
|
||
"- Load hourly OHLCV and 8-hourly premium index data via the canonical loaders.\n",
|
||
"- Document premium-index units (decimal, multiply by 100 for percent).\n",
|
||
"- Quantify cross-frequency join coverage between OHLCV and premium data.\n",
|
||
"- Run OHLC invariant checks and inspect for time-stamp gaps.\n",
|
||
"\n",
|
||
"## Book Reference\n",
|
||
"\n",
|
||
"Chapter 2 §2.2 (asset-class market data landscape — digital assets).\n",
|
||
"\n",
|
||
"## Prerequisites\n",
|
||
"\n",
|
||
"- Familiarity with daily OHLCV equity data (`01_us_equities_eda`).\n",
|
||
"- The Binance Futures parquet at `$ML4T_DATA_PATH/crypto/` (OHLCV + premium).\n",
|
||
"- Methodology continues in `11_crypto_premium_analysis`; the case-study\n",
|
||
" pipeline lives under `case_studies/crypto_perps_funding/`."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 1,
|
||
"id": "f2410448",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:33:28.980599Z",
|
||
"iopub.status.busy": "2026-06-13T02:33:28.980491Z",
|
||
"iopub.status.idle": "2026-06-13T02:33:30.219451Z",
|
||
"shell.execute_reply": "2026-06-13T02:33:30.218950Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 1.241442,
|
||
"end_time": "2026-06-13T02:33:30.220191+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:33:28.978749+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"\"\"\"Crypto Perps EDA — hourly OHLCV and premium index exploration.\"\"\"\n",
|
||
"\n",
|
||
"import polars as pl\n",
|
||
"\n",
|
||
"from data import load_crypto_perps, load_crypto_premium\n",
|
||
"from utils.data_quality import check_ohlc_invariants, per_asset_stats"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 2,
|
||
"id": "f028d40b",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:33:30.223486Z",
|
||
"iopub.status.busy": "2026-06-13T02:33:30.223403Z",
|
||
"iopub.status.idle": "2026-06-13T02:33:30.225251Z",
|
||
"shell.execute_reply": "2026-06-13T02:33:30.224946Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.004413,
|
||
"end_time": "2026-06-13T02:33:30.225693+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:33:30.221280+00:00",
|
||
"status": "completed"
|
||
},
|
||
"tags": [
|
||
"parameters"
|
||
]
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"MAX_SYMBOLS = 0 # 0 = all symbols"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "62781cf4",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.000842,
|
||
"end_time": "2026-06-13T02:33:30.227452+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:33:30.226610+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"## 1. Load and Inspect OHLCV\n",
|
||
"\n",
|
||
"Hourly OHLCV data from Binance Futures for 19 cryptocurrencies.\n",
|
||
"Trading is 24/7 (8,760 hours/year vs 252 days for equities)."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 3,
|
||
"id": "b3bc2125",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:33:30.229623Z",
|
||
"iopub.status.busy": "2026-06-13T02:33:30.229559Z",
|
||
"iopub.status.idle": "2026-06-13T02:33:30.269019Z",
|
||
"shell.execute_reply": "2026-06-13T02:33:30.268500Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.040998,
|
||
"end_time": "2026-06-13T02:33:30.269305+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:33:30.228307+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"=== OHLCV Dataset ===\n",
|
||
"Shape: (866484, 7)\n",
|
||
"Columns: ['timestamp', 'open', 'high', 'low', 'close', 'volume', 'symbol']\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"ohlcv = load_crypto_perps(frequency=\"1h\")\n",
|
||
"\n",
|
||
"print(\"=== OHLCV Dataset ===\")\n",
|
||
"print(f\"Shape: {ohlcv.shape}\")\n",
|
||
"print(f\"Columns: {ohlcv.columns}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 4,
|
||
"id": "599855a3",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:33:30.271848Z",
|
||
"iopub.status.busy": "2026-06-13T02:33:30.271778Z",
|
||
"iopub.status.idle": "2026-06-13T02:33:30.274032Z",
|
||
"shell.execute_reply": "2026-06-13T02:33:30.273658Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.003918,
|
||
"end_time": "2026-06-13T02:33:30.274317+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:33:30.270399+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"\n",
|
||
"Schema:\n",
|
||
" timestamp: Datetime(time_unit='ms', time_zone='UTC')\n",
|
||
" open: Float64\n",
|
||
" high: Float64\n",
|
||
" low: Float64\n",
|
||
" close: Float64\n",
|
||
" volume: Float64\n",
|
||
" symbol: String\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Schema overview\n",
|
||
"print(\"\\nSchema:\")\n",
|
||
"for col, dtype in ohlcv.schema.items():\n",
|
||
" print(f\" {col}: {dtype}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "51d44bf8",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.000888,
|
||
"end_time": "2026-06-13T02:33:30.276173+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:33:30.275285+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"## 2. Coverage Summary"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 5,
|
||
"id": "5c9db57a",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:33:30.278342Z",
|
||
"iopub.status.busy": "2026-06-13T02:33:30.278276Z",
|
||
"iopub.status.idle": "2026-06-13T02:33:30.285289Z",
|
||
"shell.execute_reply": "2026-06-13T02:33:30.284906Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.00843,
|
||
"end_time": "2026-06-13T02:33:30.285512+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:33:30.277082+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"=== Coverage ===\n",
|
||
"Number of symbols: 19\n",
|
||
"\n",
|
||
"Symbols: AAVEUSDT, ADAUSDT, APTUSDT, ATOMUSDT, AVAXUSDT, BNBUSDT, BTCUSDT, COMPUSDT, DOGEUSDT, DOTUSDT, ETHUSDT, INJUSDT, LINKUSDT, MKRUSDT, NEARUSDT, SOLUSDT, SUIUSDT, UNIUSDT, XRPUSDT\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Symbols and date range\n",
|
||
"symbols = ohlcv[\"symbol\"].unique().sort().to_list()\n",
|
||
"print(\"=== Coverage ===\")\n",
|
||
"print(f\"Number of symbols: {len(symbols)}\")\n",
|
||
"print(f\"\\nSymbols: {', '.join(symbols)}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 6,
|
||
"id": "92032764",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:33:30.288070Z",
|
||
"iopub.status.busy": "2026-06-13T02:33:30.287999Z",
|
||
"iopub.status.idle": "2026-06-13T02:33:30.296853Z",
|
||
"shell.execute_reply": "2026-06-13T02:33:30.296505Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.010672,
|
||
"end_time": "2026-06-13T02:33:30.297260+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:33:30.286588+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"\n",
|
||
"Date range: 2020-01-01 00:00:00+00:00 to 2025-12-31 23:00:00+00:00\n",
|
||
"Unique hours: 52,608\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Overall date range\n",
|
||
"date_range = ohlcv.select(\n",
|
||
" [\n",
|
||
" pl.col(\"timestamp\").min().alias(\"start\"),\n",
|
||
" pl.col(\"timestamp\").max().alias(\"end\"),\n",
|
||
" pl.col(\"timestamp\").n_unique().alias(\"unique_hours\"),\n",
|
||
" ]\n",
|
||
")\n",
|
||
"\n",
|
||
"print(f\"\\nDate range: {date_range['start'][0]} to {date_range['end'][0]}\")\n",
|
||
"print(f\"Unique hours: {date_range['unique_hours'][0]:,}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 7,
|
||
"id": "c9658965",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:33:30.300212Z",
|
||
"iopub.status.busy": "2026-06-13T02:33:30.300078Z",
|
||
"iopub.status.idle": "2026-06-13T02:33:30.310731Z",
|
||
"shell.execute_reply": "2026-06-13T02:33:30.310442Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.012868,
|
||
"end_time": "2026-06-13T02:33:30.311341+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:33:30.298473+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"\n",
|
||
"Symbol Statistics (top 5 by volume):\n"
|
||
]
|
||
},
|
||
{
|
||
"data": {
|
||
"text/html": [
|
||
"<div><style>\n",
|
||
".dataframe > thead > tr,\n",
|
||
".dataframe > tbody > tr {\n",
|
||
" text-align: right;\n",
|
||
" white-space: pre-wrap;\n",
|
||
"}\n",
|
||
"</style>\n",
|
||
"<small>shape: (5, 6)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>symbol</th><th>rows</th><th>start</th><th>end</th><th>avg_price</th><th>avg_volume</th></tr><tr><td>str</td><td>u32</td><td>datetime[ms, UTC]</td><td>datetime[ms, UTC]</td><td>f64</td><td>f64</td></tr></thead><tbody><tr><td>"DOGEUSDT"</td><td>48015</td><td>2020-07-10 09:00:00 UTC</td><td>2025-12-31 23:00:00 UTC</td><td>0.137195</td><td>2.7592e8</td></tr><tr><td>"XRPUSDT"</td><td>52360</td><td>2020-01-06 08:00:00 UTC</td><td>2025-12-31 23:00:00 UTC</td><td>0.89747</td><td>5.2917e7</td></tr><tr><td>"ADAUSDT"</td><td>51880</td><td>2020-01-31 08:00:00 UTC</td><td>2025-12-31 23:00:00 UTC</td><td>0.644055</td><td>3.0205e7</td></tr><tr><td>"SUIUSDT"</td><td>23360</td><td>2023-05-03 16:00:00 UTC</td><td>2025-12-31 23:00:00 UTC</td><td>1.917873</td><td>1.2587e7</td></tr><tr><td>"NEARUSDT"</td><td>45568</td><td>2020-10-15 08:00:00 UTC</td><td>2025-12-31 23:00:00 UTC</td><td>4.297627</td><td>2.1940e6</td></tr></tbody></table></div>"
|
||
],
|
||
"text/plain": [
|
||
"shape: (5, 6)\n",
|
||
"┌──────────┬───────┬─────────────────────────┬─────────────────────────┬───────────┬────────────┐\n",
|
||
"│ symbol ┆ rows ┆ start ┆ end ┆ avg_price ┆ avg_volume │\n",
|
||
"│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │\n",
|
||
"│ str ┆ u32 ┆ datetime[ms, UTC] ┆ datetime[ms, UTC] ┆ f64 ┆ f64 │\n",
|
||
"╞══════════╪═══════╪═════════════════════════╪═════════════════════════╪═══════════╪════════════╡\n",
|
||
"│ DOGEUSDT ┆ 48015 ┆ 2020-07-10 09:00:00 UTC ┆ 2025-12-31 23:00:00 UTC ┆ 0.137195 ┆ 2.7592e8 │\n",
|
||
"│ XRPUSDT ┆ 52360 ┆ 2020-01-06 08:00:00 UTC ┆ 2025-12-31 23:00:00 UTC ┆ 0.89747 ┆ 5.2917e7 │\n",
|
||
"│ ADAUSDT ┆ 51880 ┆ 2020-01-31 08:00:00 UTC ┆ 2025-12-31 23:00:00 UTC ┆ 0.644055 ┆ 3.0205e7 │\n",
|
||
"│ SUIUSDT ┆ 23360 ┆ 2023-05-03 16:00:00 UTC ┆ 2025-12-31 23:00:00 UTC ┆ 1.917873 ┆ 1.2587e7 │\n",
|
||
"│ NEARUSDT ┆ 45568 ┆ 2020-10-15 08:00:00 UTC ┆ 2025-12-31 23:00:00 UTC ┆ 4.297627 ┆ 2.1940e6 │\n",
|
||
"└──────────┴───────┴─────────────────────────┴─────────────────────────┴───────────┴────────────┘"
|
||
]
|
||
},
|
||
"execution_count": 7,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"# Per-symbol statistics\n",
|
||
"symbol_stats = per_asset_stats(\n",
|
||
" ohlcv,\n",
|
||
" time_col=\"timestamp\",\n",
|
||
" asset_col=\"symbol\",\n",
|
||
" price_col=\"close\",\n",
|
||
" volume_col=\"volume\",\n",
|
||
")\n",
|
||
"\n",
|
||
"print(\"\\nSymbol Statistics (top 5 by volume):\")\n",
|
||
"symbol_stats.sort(\"avg_volume\", descending=True).head(5)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "74c26d5a",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.001876,
|
||
"end_time": "2026-06-13T02:33:30.315436+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:33:30.313560+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"## 3. Premium Index Data\n",
|
||
"\n",
|
||
"The premium index measures the spread between perpetual futures and spot prices:\n",
|
||
"\n",
|
||
"**Premium = (Perpetual Price - Spot Price) / Spot Price**\n",
|
||
"\n",
|
||
"- Positive premium: Futures above spot (bullish sentiment)\n",
|
||
"- Negative premium: Futures below spot (bearish sentiment)\n",
|
||
"\n",
|
||
"### Units\n",
|
||
"\n",
|
||
"Premium values are stored as **decimals** (0.001 = 0.1%). When displaying,\n",
|
||
"multiply by 100 for percentage representation."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 8,
|
||
"id": "5bd439f9",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:33:30.318383Z",
|
||
"iopub.status.busy": "2026-06-13T02:33:30.318294Z",
|
||
"iopub.status.idle": "2026-06-13T02:33:30.330166Z",
|
||
"shell.execute_reply": "2026-06-13T02:33:30.329851Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.013671,
|
||
"end_time": "2026-06-13T02:33:30.330466+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:33:30.316795+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"=== Premium Dataset ===\n",
|
||
"Shape: (107839, 6)\n",
|
||
"Columns: ['timestamp', 'symbol', 'premium_index_open', 'premium_index_high', 'premium_index_low', 'premium_index_close']\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"premium = load_crypto_premium(frequency=\"8h\")\n",
|
||
"print(\"=== Premium Dataset ===\")\n",
|
||
"print(f\"Shape: {premium.shape}\")\n",
|
||
"print(f\"Columns: {premium.columns}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 9,
|
||
"id": "ac1c5ece",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:33:30.333347Z",
|
||
"iopub.status.busy": "2026-06-13T02:33:30.333260Z",
|
||
"iopub.status.idle": "2026-06-13T02:33:30.336126Z",
|
||
"shell.execute_reply": "2026-06-13T02:33:30.335774Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.004926,
|
||
"end_time": "2026-06-13T02:33:30.336661+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:33:30.331735+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"\n",
|
||
"Premium Range (decimals):\n",
|
||
" Mean: -0.000111 (-0.0111%)\n",
|
||
" Std: 0.001137 (0.1137%)\n",
|
||
" Min: -0.191547 (-19.1547%)\n",
|
||
" Max: 0.012701 (1.2701%)\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Premium range — values are decimals, not percentages\n",
|
||
"premium_range = premium.select(\n",
|
||
" [\n",
|
||
" pl.col(\"premium_index_close\").min().alias(\"min\"),\n",
|
||
" pl.col(\"premium_index_close\").max().alias(\"max\"),\n",
|
||
" pl.col(\"premium_index_close\").mean().alias(\"mean\"),\n",
|
||
" pl.col(\"premium_index_close\").std().alias(\"std\"),\n",
|
||
" ]\n",
|
||
")\n",
|
||
"\n",
|
||
"print(\"\\nPremium Range (decimals):\")\n",
|
||
"print(f\" Mean: {premium_range['mean'][0]:.6f} ({premium_range['mean'][0] * 100:.4f}%)\")\n",
|
||
"print(f\" Std: {premium_range['std'][0]:.6f} ({premium_range['std'][0] * 100:.4f}%)\")\n",
|
||
"print(f\" Min: {premium_range['min'][0]:.6f} ({premium_range['min'][0] * 100:.4f}%)\")\n",
|
||
"print(f\" Max: {premium_range['max'][0]:.6f} ({premium_range['max'][0] * 100:.4f}%)\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "bd327365",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.001018,
|
||
"end_time": "2026-06-13T02:33:30.338980+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:33:30.337962+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"## 4. Data Quality"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 10,
|
||
"id": "b2303360",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:33:30.341624Z",
|
||
"iopub.status.busy": "2026-06-13T02:33:30.341557Z",
|
||
"iopub.status.idle": "2026-06-13T02:33:30.349627Z",
|
||
"shell.execute_reply": "2026-06-13T02:33:30.349335Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.009977,
|
||
"end_time": "2026-06-13T02:33:30.350085+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:33:30.340108+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"=== OHLC Invariants ===\n",
|
||
" [OK] high_gte_low: 100.00%\n",
|
||
" [OK] high_gte_open: 100.00%\n",
|
||
" [OK] high_gte_close: 100.00%\n",
|
||
" [OK] low_lte_open: 100.00%\n",
|
||
" [OK] low_lte_close: 100.00%\n",
|
||
" [OK] volume_non_negative: 100.00%\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# OHLC invariants\n",
|
||
"invariants = check_ohlc_invariants(ohlcv)\n",
|
||
"print(\"=== OHLC Invariants ===\")\n",
|
||
"for row in invariants.iter_rows(named=True):\n",
|
||
" status = \"[OK]\" if row[\"valid_pct\"] >= 99.99 else \"[WARN]\"\n",
|
||
" print(f\" {status} {row['check']}: {row['valid_pct']:.2f}%\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 11,
|
||
"id": "81337cc6",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:33:30.352943Z",
|
||
"iopub.status.busy": "2026-06-13T02:33:30.352862Z",
|
||
"iopub.status.idle": "2026-06-13T02:33:30.355138Z",
|
||
"shell.execute_reply": "2026-06-13T02:33:30.354906Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.004193,
|
||
"end_time": "2026-06-13T02:33:30.355537+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:33:30.351344+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"\n",
|
||
"Null values: OHLCV=0, Premium=0\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Check for nulls\n",
|
||
"ohlcv_nulls = ohlcv.null_count().sum_horizontal()[0]\n",
|
||
"premium_nulls = premium.null_count().sum_horizontal()[0]\n",
|
||
"print(f\"\\nNull values: OHLCV={ohlcv_nulls}, Premium={premium_nulls}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 12,
|
||
"id": "ed1e0266",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:33:30.358207Z",
|
||
"iopub.status.busy": "2026-06-13T02:33:30.358135Z",
|
||
"iopub.status.idle": "2026-06-13T02:33:30.363600Z",
|
||
"shell.execute_reply": "2026-06-13T02:33:30.363206Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.007243,
|
||
"end_time": "2026-06-13T02:33:30.363925+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:33:30.356682+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"\n",
|
||
"Gaps > 1 hour in BTC: 0\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Check for gaps > 1 hour (use BTC as reference)\n",
|
||
"btc = ohlcv.filter(pl.col(\"symbol\") == \"BTCUSDT\").sort(\"timestamp\")\n",
|
||
"btc_gaps = btc.with_columns(pl.col(\"timestamp\").diff().dt.total_hours().alias(\"hours_diff\")).filter(\n",
|
||
" pl.col(\"hours_diff\") > 1\n",
|
||
")\n",
|
||
"\n",
|
||
"print(f\"\\nGaps > 1 hour in BTC: {len(btc_gaps)}\")\n",
|
||
"if len(btc_gaps) > 0:\n",
|
||
" print(\"(Small gaps expected during exchange maintenance)\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "ffa0b798",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.001072,
|
||
"end_time": "2026-06-13T02:33:30.366245+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:33:30.365173+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"## 5. Joining OHLCV and Premium\n",
|
||
"\n",
|
||
"Use left join to preserve all OHLCV rows and identify missing premium coverage."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 13,
|
||
"id": "c311a381",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:33:30.369052Z",
|
||
"iopub.status.busy": "2026-06-13T02:33:30.368962Z",
|
||
"iopub.status.idle": "2026-06-13T02:33:30.412049Z",
|
||
"shell.execute_reply": "2026-06-13T02:33:30.411644Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.045382,
|
||
"end_time": "2026-06-13T02:33:30.412749+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:33:30.367367+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"=== Join Coverage ===\n",
|
||
"OHLCV rows: 866,484\n",
|
||
"Premium rows: 107,839\n",
|
||
"Combined rows: 866,484\n",
|
||
"Missing premium: 758,716 (87.56%)\n",
|
||
"Coverage: 12.44%\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Left join to identify missing premium data\n",
|
||
"combined = ohlcv.join(premium, on=[\"timestamp\", \"symbol\"], how=\"left\")\n",
|
||
"\n",
|
||
"# Coverage analysis\n",
|
||
"total_rows = len(combined)\n",
|
||
"missing_premium = combined.filter(pl.col(\"premium_index_close\").is_null()).height\n",
|
||
"coverage_pct = (total_rows - missing_premium) / total_rows * 100\n",
|
||
"\n",
|
||
"print(\"=== Join Coverage ===\")\n",
|
||
"print(f\"OHLCV rows: {len(ohlcv):,}\")\n",
|
||
"print(f\"Premium rows: {len(premium):,}\")\n",
|
||
"print(f\"Combined rows: {total_rows:,}\")\n",
|
||
"print(f\"Missing premium: {missing_premium:,} ({100 - coverage_pct:.2f}%)\")\n",
|
||
"print(f\"Coverage: {coverage_pct:.2f}%\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 14,
|
||
"id": "011f6a3b",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T02:33:30.417422Z",
|
||
"iopub.status.busy": "2026-06-13T02:33:30.417324Z",
|
||
"iopub.status.idle": "2026-06-13T02:33:30.434943Z",
|
||
"shell.execute_reply": "2026-06-13T02:33:30.434671Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.020165,
|
||
"end_time": "2026-06-13T02:33:30.435349+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:33:30.415184+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Missing premium by symbol (top 5):\n"
|
||
]
|
||
},
|
||
{
|
||
"data": {
|
||
"text/html": [
|
||
"<div><style>\n",
|
||
".dataframe > thead > tr,\n",
|
||
".dataframe > tbody > tr {\n",
|
||
" text-align: right;\n",
|
||
" white-space: pre-wrap;\n",
|
||
"}\n",
|
||
"</style>\n",
|
||
"<small>shape: (5, 2)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>symbol</th><th>len</th></tr><tr><td>str</td><td>u32</td></tr></thead><tbody><tr><td>"BTCUSDT"</td><td>46053</td></tr><tr><td>"ETHUSDT"</td><td>46053</td></tr><tr><td>"XRPUSDT"</td><td>45833</td></tr><tr><td>"LINKUSDT"</td><td>45710</td></tr><tr><td>"ADAUSDT"</td><td>45416</td></tr></tbody></table></div>"
|
||
],
|
||
"text/plain": [
|
||
"shape: (5, 2)\n",
|
||
"┌──────────┬───────┐\n",
|
||
"│ symbol ┆ len │\n",
|
||
"│ --- ┆ --- │\n",
|
||
"│ str ┆ u32 │\n",
|
||
"╞══════════╪═══════╡\n",
|
||
"│ BTCUSDT ┆ 46053 │\n",
|
||
"│ ETHUSDT ┆ 46053 │\n",
|
||
"│ XRPUSDT ┆ 45833 │\n",
|
||
"│ LINKUSDT ┆ 45710 │\n",
|
||
"│ ADAUSDT ┆ 45416 │\n",
|
||
"└──────────┴───────┘"
|
||
]
|
||
},
|
||
"execution_count": 14,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"# Where does the missing premium concentrate?\n",
|
||
"missing_by_symbol = (\n",
|
||
" combined.filter(pl.col(\"premium_index_close\").is_null())\n",
|
||
" .group_by(\"symbol\")\n",
|
||
" .len()\n",
|
||
" .sort(\"len\", descending=True)\n",
|
||
")\n",
|
||
"print(\"Missing premium by symbol (top 5):\")\n",
|
||
"missing_by_symbol.head(5)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "869f500e",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.001191,
|
||
"end_time": "2026-06-13T02:33:30.437914+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T02:33:30.436723+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"## Key Takeaways\n",
|
||
"\n",
|
||
"1. **24/7 trading**: Crypto runs continuously — 52,608 unique hours across\n",
|
||
" six full years (2020-01-01 to 2025-12-31), ~8,760 hours/year for the\n",
|
||
" longest-history symbols.\n",
|
||
"2. **Universe**: 19 perpetual contracts span 866,484 OHLCV bars; symbol\n",
|
||
" coverage is non-uniform because contracts list at different dates.\n",
|
||
"3. **Premium units**: Stored as decimals (0.001 = 0.1%); always multiply by\n",
|
||
" 100 for percentage display in figures or text.\n",
|
||
"4. **Frequency mismatch**: OHLCV is hourly, premium is 8-hourly, so a left\n",
|
||
" join lands ~12.4% premium coverage on the hourly grid by design.\n",
|
||
"5. **Clean data**: OHLC invariants hold for 100% of records on this\n",
|
||
" snapshot; BTC has zero gaps > 1 hour.\n",
|
||
"\n",
|
||
"## Next Steps\n",
|
||
"\n",
|
||
"- `11_crypto_premium_analysis`: Premium dynamics, basis seasonality, and\n",
|
||
" alignment to the 8-hourly funding cadence.\n",
|
||
"- Chapter 8: Feature engineering for premium signals\n",
|
||
" (`case_studies/crypto_perps_funding/03_financial_features.py`).\n",
|
||
"- Chapter 16: Backtests for the funding-arbitrage case study."
|
||
]
|
||
}
|
||
],
|
||
"metadata": {
|
||
"jupytext": {
|
||
"cell_metadata_filter": "tags,-all"
|
||
},
|
||
"kernelspec": {
|
||
"display_name": "Python 3 (ipykernel)",
|
||
"language": "python",
|
||
"name": "python3"
|
||
},
|
||
"language_info": {
|
||
"codemirror_mode": {
|
||
"name": "ipython",
|
||
"version": 3
|
||
},
|
||
"file_extension": ".py",
|
||
"mimetype": "text/x-python",
|
||
"name": "python",
|
||
"nbconvert_exporter": "python",
|
||
"pygments_lexer": "ipython3",
|
||
"version": "3.14.3"
|
||
},
|
||
"papermill": {
|
||
"default_parameters": {},
|
||
"duration": 2.819325,
|
||
"end_time": "2026-06-13T02:33:31.153715+00:00",
|
||
"environment_variables": {},
|
||
"exception": null,
|
||
"input_path": "02_financial_data_universe/10_crypto_perps_eda.ipynb",
|
||
"output_path": "02_financial_data_universe/10_crypto_perps_eda.ipynb",
|
||
"parameters": {},
|
||
"start_time": "2026-06-13T02:33:28.334390+00:00",
|
||
"version": "2.7.0"
|
||
},
|
||
"ml4t_provenance": {
|
||
"source_py_blob": "ad9e3464a73b7086edefb3b2413e3c50096de365",
|
||
"executed_at": "2026-06-13T02:33:31.465291+00:00",
|
||
"executor": "cpu-local",
|
||
"production": true,
|
||
"parameters": {},
|
||
"notes": "executed-state finalization batch (2026-06-13 run)"
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 5
|
||
}
|