wehub-resource-sync
232 lines
7.2 KiB
Python
232 lines
7.2 KiB
Python
# ---
|
||
# jupyter:
|
||
# jupytext:
|
||
# cell_metadata_filter: tags,-all
|
||
# text_representation:
|
||
# extension: .py
|
||
# format_name: percent
|
||
# format_version: '1.3'
|
||
# jupytext_version: 1.19.3
|
||
# kernelspec:
|
||
# display_name: Python 3 (ipykernel)
|
||
# language: python
|
||
# name: python3
|
||
# ---
|
||
|
||
# %% [markdown]
|
||
# # Crypto Perps — Exploratory Data Analysis
|
||
#
|
||
# **Docker image**: `ml4t`
|
||
#
|
||
# ## Purpose
|
||
#
|
||
# Profile the Binance Futures hourly OHLCV dataset for 19 perpetual contracts
|
||
# alongside the 8-hourly Premium Index that captures the perpetual–spot
|
||
# basis. The notebook anchors data shape, units, coverage, and OHLC integrity
|
||
# before the strategy work begins in Chapter 6.
|
||
#
|
||
# ## Learning Objectives
|
||
#
|
||
# - Load hourly OHLCV and 8-hourly premium index data via the canonical loaders.
|
||
# - Document premium-index units (decimal, multiply by 100 for percent).
|
||
# - Quantify cross-frequency join coverage between OHLCV and premium data.
|
||
# - Run OHLC invariant checks and inspect for time-stamp gaps.
|
||
#
|
||
# ## Book Reference
|
||
#
|
||
# Chapter 2 §2.2 (asset-class market data landscape — digital assets).
|
||
#
|
||
# ## Prerequisites
|
||
#
|
||
# - Familiarity with daily OHLCV equity data (`01_us_equities_eda`).
|
||
# - The Binance Futures parquet at `$ML4T_DATA_PATH/crypto/` (OHLCV + premium).
|
||
# - Methodology continues in `11_crypto_premium_analysis`; the case-study
|
||
# pipeline lives under `case_studies/crypto_perps_funding/`.
|
||
|
||
# %%
|
||
"""Crypto Perps EDA — hourly OHLCV and premium index exploration."""
|
||
|
||
import polars as pl
|
||
|
||
from data import load_crypto_perps, load_crypto_premium
|
||
from utils.data_quality import check_ohlc_invariants, per_asset_stats
|
||
|
||
# %% tags=["parameters"]
|
||
MAX_SYMBOLS = 0 # 0 = all symbols
|
||
|
||
# %% [markdown]
|
||
# ## 1. Load and Inspect OHLCV
|
||
#
|
||
# Hourly OHLCV data from Binance Futures for 19 cryptocurrencies.
|
||
# Trading is 24/7 (8,760 hours/year vs 252 days for equities).
|
||
|
||
# %%
|
||
ohlcv = load_crypto_perps(frequency="1h")
|
||
|
||
print("=== OHLCV Dataset ===")
|
||
print(f"Shape: {ohlcv.shape}")
|
||
print(f"Columns: {ohlcv.columns}")
|
||
|
||
# %%
|
||
# Schema overview
|
||
print("\nSchema:")
|
||
for col, dtype in ohlcv.schema.items():
|
||
print(f" {col}: {dtype}")
|
||
|
||
# %% [markdown]
|
||
# ## 2. Coverage Summary
|
||
|
||
# %%
|
||
# Symbols and date range
|
||
symbols = ohlcv["symbol"].unique().sort().to_list()
|
||
print("=== Coverage ===")
|
||
print(f"Number of symbols: {len(symbols)}")
|
||
print(f"\nSymbols: {', '.join(symbols)}")
|
||
|
||
# %%
|
||
# Overall date range
|
||
date_range = ohlcv.select(
|
||
[
|
||
pl.col("timestamp").min().alias("start"),
|
||
pl.col("timestamp").max().alias("end"),
|
||
pl.col("timestamp").n_unique().alias("unique_hours"),
|
||
]
|
||
)
|
||
|
||
print(f"\nDate range: {date_range['start'][0]} to {date_range['end'][0]}")
|
||
print(f"Unique hours: {date_range['unique_hours'][0]:,}")
|
||
|
||
# %%
|
||
# Per-symbol statistics
|
||
symbol_stats = per_asset_stats(
|
||
ohlcv,
|
||
time_col="timestamp",
|
||
asset_col="symbol",
|
||
price_col="close",
|
||
volume_col="volume",
|
||
)
|
||
|
||
print("\nSymbol Statistics (top 5 by volume):")
|
||
symbol_stats.sort("avg_volume", descending=True).head(5)
|
||
|
||
# %% [markdown]
|
||
# ## 3. Premium Index Data
|
||
#
|
||
# The premium index measures the spread between perpetual futures and spot prices:
|
||
#
|
||
# **Premium = (Perpetual Price - Spot Price) / Spot Price**
|
||
#
|
||
# - Positive premium: Futures above spot (bullish sentiment)
|
||
# - Negative premium: Futures below spot (bearish sentiment)
|
||
#
|
||
# ### Units
|
||
#
|
||
# Premium values are stored as **decimals** (0.001 = 0.1%). When displaying,
|
||
# multiply by 100 for percentage representation.
|
||
|
||
# %%
|
||
premium = load_crypto_premium(frequency="8h")
|
||
print("=== Premium Dataset ===")
|
||
print(f"Shape: {premium.shape}")
|
||
print(f"Columns: {premium.columns}")
|
||
|
||
# %%
|
||
# Premium range — values are decimals, not percentages
|
||
premium_range = premium.select(
|
||
[
|
||
pl.col("premium_index_close").min().alias("min"),
|
||
pl.col("premium_index_close").max().alias("max"),
|
||
pl.col("premium_index_close").mean().alias("mean"),
|
||
pl.col("premium_index_close").std().alias("std"),
|
||
]
|
||
)
|
||
|
||
print("\nPremium Range (decimals):")
|
||
print(f" Mean: {premium_range['mean'][0]:.6f} ({premium_range['mean'][0] * 100:.4f}%)")
|
||
print(f" Std: {premium_range['std'][0]:.6f} ({premium_range['std'][0] * 100:.4f}%)")
|
||
print(f" Min: {premium_range['min'][0]:.6f} ({premium_range['min'][0] * 100:.4f}%)")
|
||
print(f" Max: {premium_range['max'][0]:.6f} ({premium_range['max'][0] * 100:.4f}%)")
|
||
|
||
# %% [markdown]
|
||
# ## 4. Data Quality
|
||
|
||
# %%
|
||
# OHLC invariants
|
||
invariants = check_ohlc_invariants(ohlcv)
|
||
print("=== OHLC Invariants ===")
|
||
for row in invariants.iter_rows(named=True):
|
||
status = "[OK]" if row["valid_pct"] >= 99.99 else "[WARN]"
|
||
print(f" {status} {row['check']}: {row['valid_pct']:.2f}%")
|
||
|
||
# %%
|
||
# Check for nulls
|
||
ohlcv_nulls = ohlcv.null_count().sum_horizontal()[0]
|
||
premium_nulls = premium.null_count().sum_horizontal()[0]
|
||
print(f"\nNull values: OHLCV={ohlcv_nulls}, Premium={premium_nulls}")
|
||
|
||
# %%
|
||
# Check for gaps > 1 hour (use BTC as reference)
|
||
btc = ohlcv.filter(pl.col("symbol") == "BTCUSDT").sort("timestamp")
|
||
btc_gaps = btc.with_columns(pl.col("timestamp").diff().dt.total_hours().alias("hours_diff")).filter(
|
||
pl.col("hours_diff") > 1
|
||
)
|
||
|
||
print(f"\nGaps > 1 hour in BTC: {len(btc_gaps)}")
|
||
if len(btc_gaps) > 0:
|
||
print("(Small gaps expected during exchange maintenance)")
|
||
|
||
# %% [markdown]
|
||
# ## 5. Joining OHLCV and Premium
|
||
#
|
||
# Use left join to preserve all OHLCV rows and identify missing premium coverage.
|
||
|
||
# %%
|
||
# Left join to identify missing premium data
|
||
combined = ohlcv.join(premium, on=["timestamp", "symbol"], how="left")
|
||
|
||
# Coverage analysis
|
||
total_rows = len(combined)
|
||
missing_premium = combined.filter(pl.col("premium_index_close").is_null()).height
|
||
coverage_pct = (total_rows - missing_premium) / total_rows * 100
|
||
|
||
print("=== Join Coverage ===")
|
||
print(f"OHLCV rows: {len(ohlcv):,}")
|
||
print(f"Premium rows: {len(premium):,}")
|
||
print(f"Combined rows: {total_rows:,}")
|
||
print(f"Missing premium: {missing_premium:,} ({100 - coverage_pct:.2f}%)")
|
||
print(f"Coverage: {coverage_pct:.2f}%")
|
||
|
||
# %%
|
||
# Where does the missing premium concentrate?
|
||
missing_by_symbol = (
|
||
combined.filter(pl.col("premium_index_close").is_null())
|
||
.group_by("symbol")
|
||
.len()
|
||
.sort("len", descending=True)
|
||
)
|
||
print("Missing premium by symbol (top 5):")
|
||
missing_by_symbol.head(5)
|
||
|
||
# %% [markdown]
|
||
# ## Key Takeaways
|
||
#
|
||
# 1. **24/7 trading**: Crypto runs continuously — 52,608 unique hours across
|
||
# six full years (2020-01-01 to 2025-12-31), ~8,760 hours/year for the
|
||
# longest-history symbols.
|
||
# 2. **Universe**: 19 perpetual contracts span 866,484 OHLCV bars; symbol
|
||
# coverage is non-uniform because contracts list at different dates.
|
||
# 3. **Premium units**: Stored as decimals (0.001 = 0.1%); always multiply by
|
||
# 100 for percentage display in figures or text.
|
||
# 4. **Frequency mismatch**: OHLCV is hourly, premium is 8-hourly, so a left
|
||
# join lands ~12.4% premium coverage on the hourly grid by design.
|
||
# 5. **Clean data**: OHLC invariants hold for 100% of records on this
|
||
# snapshot; BTC has zero gaps > 1 hour.
|
||
#
|
||
# ## Next Steps
|
||
#
|
||
# - `11_crypto_premium_analysis`: Premium dynamics, basis seasonality, and
|
||
# alignment to the 8-hourly funding cadence.
|
||
# - Chapter 8: Feature engineering for premium signals
|
||
# (`case_studies/crypto_perps_funding/03_financial_features.py`).
|
||
# - Chapter 16: Backtests for the funding-arbitrage case study.
|