Files
stefan-jansen--machine-lear…/02_financial_data_universe/11_crypto_premium_analysis.py
T
2026-07-13 13:26:28 +08:00

595 lines
19 KiB
Python
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# ---
# jupyter:
# jupytext:
# cell_metadata_filter: tags,-all
# formats: ipynb,py:percent
# text_representation:
# extension: .py
# format_name: percent
# format_version: '1.3'
# jupytext_version: 1.19.3
# kernelspec:
# display_name: Python 3 (ipykernel)
# language: python
# name: python3
# ---
# %% [markdown]
# # Crypto Premium Index: Funding Rate Arbitrage Data
#
# **Docker image**: `ml4t`
#
# ## Purpose
# Explore the Binance perpetual-futures premium index — the per-period deviation between
# perpetual and spot prices that determines the funding rate paid every 8 hours. The
# notebook profiles 19 USDT-margined perpetuals from 2020-01 to 2025-12 and turns the
# raw premium series into estimated funding APY for the funding-arbitrage case study.
#
# ## Learning Objectives
# - Load the 8-hour premium-index panel and read its schema.
# - Characterize the distribution and time-series behavior of BTC premium.
# - Compare premium volatility across majors and altcoins.
# - Translate premium into Binance's clamped funding rate and an annualized return.
#
# ## Book reference
# Chapter 2, §2.2 (asset-class market data — crypto datasets). The funding-arbitrage
# case study built on this dataset lives in `case_studies/crypto_perps_funding/`.
#
# ## Prerequisites
# - Crypto perpetual + premium parquet files materialized under `ML4T_DATA_PATH`.
# - Loader `data.load_crypto_premium`.
#
# ---
# %%
"""Crypto Premium Index — Funding rate arbitrage data exploration."""
import numpy as np
import plotly.express as px
import plotly.graph_objects as go
import polars as pl
from plotly.subplots import make_subplots
from data import load_crypto_premium
# %% tags=["parameters"]
# Production defaults — Papermill injects overrides for CI
# %% [markdown]
# ---
#
# ## Section 1: Understanding the Premium Index
#
# ### What is the Premium Index?
#
# The **Premium Index** measures the deviation between perpetual futures prices and spot prices:
#
# $$\text{Premium Index} = \frac{\text{Perpetual Price} - \text{Spot Price}}{\text{Spot Price}}$$
#
# ### Key Properties:
#
# 1. **Positive Premium**: Perpetual > Spot → Longs pay Shorts (bullish sentiment)
# 2. **Negative Premium**: Perpetual < Spot → Shorts pay Longs (bearish sentiment)
# 3. **Funding Rate**: Derived from premium index, paid every 8 hours on Binance
#
# ### Arbitrage Opportunity
#
# When premium is significantly positive:
# - **Long Spot** + **Short Perpetual** = Collect funding payments
# - Market-neutral position captures the funding rate
#
# When premium is significantly negative:
# - **Short Spot** + **Long Perpetual** = Collect funding payments
# %%
# Load the combined premium index data
premium_df = load_crypto_premium(frequency="8h")
print(f"Total rows: {len(premium_df):,}")
print(f"Columns: {premium_df.columns}")
print("\nSchema:")
for col, dtype in premium_df.schema.items():
print(f" {col}: {dtype}")
# %%
# Overview by asset
symbol_stats = (
premium_df.group_by("symbol")
.agg(
[
pl.col("timestamp").min().alias("start"),
pl.col("timestamp").max().alias("end"),
pl.len().alias("rows"),
pl.col("premium_index_close").mean().alias("avg_premium"),
pl.col("premium_index_close").std().alias("std_premium"),
]
)
.sort("rows", descending=True)
)
symbol_stats
# %%
# Sample data - BTC premium index
btc_premium = premium_df.filter(pl.col("symbol") == "BTCUSDT").sort("timestamp")
print(f"BTC Premium Index: {len(btc_premium):,} 8h observations")
print(f"Date range: {btc_premium['timestamp'].min()} to {btc_premium['timestamp'].max()}")
btc_premium.head(10)
# %% [markdown]
# ---
#
# ## Section 2: Premium Index Distribution
#
# Understanding the distribution of premium values is crucial for:
# 1. Setting entry/exit thresholds for arbitrage
# 2. Risk management (tail events)
# 3. Comparing opportunities across assets
# %%
# BTC Premium distribution
btc_close = btc_premium["premium_index_close"].to_numpy()
# Convert to basis points for readability
btc_close_bps = btc_close * 10000
fig = go.Figure()
fig.add_trace(
go.Histogram(
x=btc_close_bps,
nbinsx=100,
name="BTC Premium",
marker_color="#F7931A",
)
)
# Add vertical lines for mean and +-2 std
mean_val = np.mean(btc_close_bps)
std_val = np.std(btc_close_bps)
fig.add_vline(
x=mean_val, line_dash="dash", line_color="red", annotation_text=f"Mean: {mean_val:.1f} bps"
)
fig.add_vline(
x=mean_val + 2 * std_val,
line_dash="dot",
line_color="green",
annotation_text=f"+2σ: {mean_val + 2 * std_val:.1f} bps",
)
fig.add_vline(
x=mean_val - 2 * std_val,
line_dash="dot",
line_color="green",
annotation_text=f"-2σ: {mean_val - 2 * std_val:.1f} bps",
)
fig.update_layout(
title="BTC Premium Index Distribution (Basis Points)",
xaxis_title="Premium Index (bps)",
yaxis_title="Frequency",
height=400,
template="plotly_white",
)
fig.show()
# %%
print("BTC Premium Statistics:")
print(f" Mean: {mean_val:.2f} bps")
print(f" Std: {std_val:.2f} bps")
print(f" Min: {np.min(btc_close_bps):.2f} bps")
print(f" Max: {np.max(btc_close_bps):.2f} bps")
print(f" Skew: {((btc_close_bps - mean_val) ** 3).mean() / std_val**3:.2f}")
# %%
# Compare premium distributions across major assets
major_symbols = ["BTCUSDT", "ETHUSDT", "SOLUSDT", "BNBUSDT"]
fig = make_subplots(rows=2, cols=2, subplot_titles=major_symbols)
colors = ["#F7931A", "#627EEA", "#00FFA3", "#F3BA2F"]
for idx, (symbol, color) in enumerate(zip(major_symbols, colors, strict=False)):
row = idx // 2 + 1
col = idx % 2 + 1
data = premium_df.filter(pl.col("symbol") == symbol)["premium_index_close"].to_numpy() * 10000
fig.add_trace(
go.Histogram(x=data, nbinsx=50, marker_color=color, name=symbol), row=row, col=col
)
fig.update_layout(
title="Premium Index Distributions (bps) - Major Cryptos",
height=500,
showlegend=False,
template="plotly_white",
)
fig.show()
# %% [markdown]
# ---
#
# ## Section 3: Time Series Analysis
#
# Premium index varies over time based on market sentiment. Let's analyze:
# 1. Long-term trends
# 2. Regime changes (bull vs bear markets)
# 3. Correlation with price movements
# %%
# BTC premium time series.
# Data is on an 8h cadence (3 obs per day), so 30 days = 90 windows.
PERIODS_PER_DAY = 3 # Binance funding interval is 8h
ROLLING_WINDOW_DAYS = 30
btc_bps = btc_premium.with_columns(
(pl.col("premium_index_close") * 10000).alias("premium_bps"),
).with_columns(
pl.col("premium_index_close")
.rolling_mean(window_size=ROLLING_WINDOW_DAYS * PERIODS_PER_DAY)
.alias("rolling_30d"),
)
# %%
# Plot raw 8h premium and 30-day rolling mean
fig = make_subplots(
rows=2,
cols=1,
shared_xaxes=True,
vertical_spacing=0.1,
subplot_titles=[
"BTC Premium Index (bps, 8h observations)",
f"{ROLLING_WINDOW_DAYS}-Day Rolling Average",
],
)
fig.add_trace(
go.Scatter(
x=btc_bps["timestamp"].to_list(),
y=btc_bps["premium_bps"].to_list(),
mode="lines",
name="8h Premium",
line=dict(color="#F7931A", width=1),
opacity=0.6,
),
row=1,
col=1,
)
fig.add_trace(
go.Scatter(
x=btc_bps["timestamp"].to_list(),
y=(btc_bps["rolling_30d"] * 10000).to_list(),
mode="lines",
name="30-Day Rolling Avg",
line=dict(color="red", width=2),
),
row=2,
col=1,
)
fig.add_hline(y=0, line_dash="dash", line_color="gray", row=1, col=1)
fig.add_hline(y=0, line_dash="dash", line_color="gray", row=2, col=1)
fig.update_layout(height=600, template="plotly_white", showlegend=False)
fig.update_yaxes(title_text="Premium (bps)", row=1, col=1)
fig.update_yaxes(title_text="Premium (bps)", row=2, col=1)
fig.show()
# %%
# Report the observed BTC range so the reader can size the y-axis.
btc_bps_series = btc_bps["premium_bps"]
print(f"BTC premium range: {btc_bps_series.min():.1f} to {btc_bps_series.max():.1f} bps")
# %%
# Identify premium regimes
btc_regimes = btc_premium.with_columns(
[
# Define regimes based on premium level
pl.when(pl.col("premium_index_close") > 0.001)
.then(pl.lit("High Premium (Bullish)"))
.when(pl.col("premium_index_close") < -0.001)
.then(pl.lit("Low Premium (Bearish)"))
.otherwise(pl.lit("Neutral"))
.alias("regime"),
# Year for grouping
pl.col("timestamp").dt.year().alias("year"),
]
)
# Regime distribution by year (counts of 8h periods, ~1095 per full year)
regime_dist = (
btc_regimes.group_by(["year", "regime"])
.agg(pl.len().alias("periods_8h"))
.sort(["year", "regime"])
)
regime_dist.pivot(on="regime", index="year", values="periods_8h").fill_null(0)
# %% [markdown]
# ---
#
# ## Section 4: Cross-Asset Premium Comparison
#
# Different cryptocurrencies have different premium dynamics:
# - **BTC/ETH**: Lower volatility, tighter premiums
# - **Altcoins**: Higher volatility, wider premium swings
#
# This affects arbitrage opportunity selection.
# %%
# Calculate premium statistics for all assets
premium_stats = (
premium_df.group_by("symbol")
.agg(
[
pl.col("premium_index_close").mean().alias("mean_premium"),
pl.col("premium_index_close").std().alias("std_premium"),
pl.col("premium_index_close").min().alias("min_premium"),
pl.col("premium_index_close").max().alias("max_premium"),
# Percentage of time premium > 10 bps (profitable arbitrage threshold)
(pl.col("premium_index_close").abs() > 0.001).mean().alias("pct_above_10bps"),
]
)
.sort("std_premium", descending=True)
)
# Convert to basis points for display
premium_stats_bps = premium_stats.with_columns(
[
(pl.col("mean_premium") * 10000).round(2).alias("mean_bps"),
(pl.col("std_premium") * 10000).round(2).alias("std_bps"),
(pl.col("min_premium") * 10000).round(2).alias("min_bps"),
(pl.col("max_premium") * 10000).round(2).alias("max_bps"),
(pl.col("pct_above_10bps") * 100).round(1).alias("pct_above_10bps"),
]
).select(["symbol", "mean_bps", "std_bps", "min_bps", "max_bps", "pct_above_10bps"])
premium_stats_bps
# %%
# Scatter: Premium volatility vs mean premium
fig = px.scatter(
premium_stats_bps.to_pandas(),
x="std_bps",
y="mean_bps",
size="pct_above_10bps",
color="symbol",
hover_name="symbol",
title="Premium Volatility vs Mean Premium",
labels={
"std_bps": "Premium Volatility (bps)",
"mean_bps": "Mean Premium (bps)",
"pct_above_10bps": "% Time > 10bps",
},
)
fig.update_layout(
height=600,
template="plotly_white",
legend=dict(
orientation="h",
yanchor="top",
y=-0.15,
xanchor="center",
x=0.5,
),
margin=dict(b=120),
)
fig.show()
print("\nInterpretation:")
print("- Top-right quadrant: High volatility, positive bias (bullish altcoins)")
print("- Larger bubbles: More arbitrage opportunities (premium often > 10bps)")
# %%
# Monthly premium heatmap
# Note: Some months have extreme values (e.g., SOL during FTX collapse at -72 bps)
# We clip the color scale at ±20 bps for better visualization of typical patterns
monthly_premium = (
premium_df.with_columns([pl.col("timestamp").dt.strftime("%Y-%m").alias("month")])
.group_by(["symbol", "month"])
.agg(pl.col("premium_index_close").mean().alias("avg_premium"))
)
# Pivot for heatmap
heatmap_data = monthly_premium.pivot(on="month", index="symbol", values="avg_premium").sort(
"symbol"
)
# Get month columns in order
month_cols = sorted([c for c in heatmap_data.columns if c != "symbol"])
assets = heatmap_data["symbol"].to_list()
# Extract values for heatmap
z_values = heatmap_data.select(month_cols).to_numpy() * 10000 # Convert to bps
# Clip color scale at ±20 bps for better visualization
COLOR_CLIP_BPS = 20
# %%
fig = go.Figure(
data=go.Heatmap(
z=z_values,
x=month_cols,
y=assets,
colorscale="RdBu",
zmid=0,
zmin=-COLOR_CLIP_BPS,
zmax=COLOR_CLIP_BPS,
colorbar=dict(title="Premium (bps)"),
)
)
fig.update_layout(
title=f"Monthly Average Premium by Asset (bps, color clipped at ±{COLOR_CLIP_BPS})",
xaxis_title="Month",
yaxis_title="Symbol",
height=600,
template="plotly_white",
)
fig.show()
# Report extremes that exceed color scale (shown as saturated colors)
extremes = (
monthly_premium.filter(pl.col("avg_premium").abs() * 10000 > COLOR_CLIP_BPS)
.with_columns((pl.col("avg_premium") * 10000).round(1).alias("avg_bps"))
.select(["symbol", "month", "avg_bps"])
.sort("avg_bps")
)
extremes
# %% [markdown]
# ---
#
# ## Section 5: Funding Rate Estimation
#
# Binance calculates funding rates from premium index every 8 hours:
#
# $$\text{Funding Rate} = \text{clamp}(\text{Premium Index}, -0.05\%, 0.05\%) + \text{Interest Rate}$$
#
# Where Interest Rate ≈ 0.01% (0.03%/day).
#
# **Annualized Return** from funding collection:
# $$\text{APY} = \text{Funding Rate} \times 3 \times 365$$
# %%
# Calculate estimated funding rates using native Polars expressions
# Formula: funding_rate = clamp(premium, -0.05%, 0.05%) + interest_rate
# Interest rate ≈ 0.01% per 8h (0.0001)
INTEREST_RATE = 0.0001
btc_funding = btc_premium.with_columns(
# Clamp premium to [-0.05%, 0.05%] and add interest rate
(pl.col("premium_index_close").clip(-0.0005, 0.0005) + INTEREST_RATE).alias("est_funding_rate"),
).with_columns(
# Annualized return: 3 funding periods/day * 365 days * 100 for percentage
(pl.col("est_funding_rate") * 3 * 365 * 100).alias("annualized_pct"),
)
avg_funding_rate = float(btc_funding["est_funding_rate"].mean())
ann_min = float(btc_funding["annualized_pct"].min())
ann_max = float(btc_funding["annualized_pct"].max())
ann_mean = float(btc_funding["annualized_pct"].mean())
print("BTC Estimated Funding Rate Analysis:")
print(f" Average funding rate (per 8h): {avg_funding_rate * 100:.4f}%")
print(f" Annualized return (avg): {ann_mean:.1f}%")
print(f" Annualized return (max): {ann_max:.1f}%")
print(f" Annualized return (min): {ann_min:.1f}%")
# %%
# Visualize annualized funding returns over time
# Note: Funding rate is clamped to ±0.05% per period, so annualized range is bounded
# to approximately ±55% (3 periods/day × 365 days × 0.05%)
fig = go.Figure()
fig.add_trace(
go.Scatter(
x=btc_funding["timestamp"].to_list(),
y=btc_funding["annualized_pct"].to_list(),
mode="lines",
name="Annualized Funding Return",
line=dict(color="#F7931A", width=1),
)
)
# Add horizontal lines for reference
fig.add_hline(y=0, line_dash="dash", line_color="gray")
fig.add_hline(y=20, line_dash="dot", line_color="green", annotation_text="20% APY")
fig.add_hline(y=-20, line_dash="dot", line_color="red", annotation_text="-20% APY")
y_padding = 10
fig.update_layout(
title="BTC Estimated Annualized Funding Return (%)",
xaxis_title="Date",
yaxis_title="Annualized Return (%)",
yaxis=dict(range=[ann_min - y_padding, ann_max + y_padding]),
height=400,
template="plotly_white",
)
fig.show()
print(f"Annualized funding return range: {ann_min:.1f}% to {ann_max:.1f}%")
# %%
# 8h periods where the estimated funding APY exceeds ±20%.
# Note: the funding-rate clamp pins per-period funding at ±0.0005 + 0.0001 interest,
# so the APY ceiling is 3 × 365 × 0.0006 × 100 ≈ 65.7% (and floor ≈ 43.8%);
# the top rows therefore all sit at the clamp.
high_conviction = btc_funding.filter(pl.col("annualized_pct").abs() > 20)
print(
f"High-conviction periods (|APY| > 20%): {len(high_conviction):,} of {len(btc_funding):,} 8h periods"
)
print(f"Share of total: {len(high_conviction) / len(btc_funding) * 100:.1f}%")
(
high_conviction.sort("annualized_pct", descending=True)
.head(10)
.select(["timestamp", "premium_index_close", "est_funding_rate", "annualized_pct"])
)
# %% [markdown]
# ---
#
# ## Section 6: Using the CryptoDataManager
#
# The ml4t-data library provides a `CryptoDataManager` for convenient access to the premium index data.
# %%
# Using the CryptoDataManager (requires ml4t-data library)
# This demonstrates the programmatic API for loading crypto data
from ml4t.data.crypto import CryptoDataManager # noqa: F401
# CryptoDataManager provides a clean API for loading crypto data
# For this notebook, we use direct parquet loading as shown above
print("CryptoDataManager API available from ml4t-data library.")
print("For this analysis, we use direct parquet loading for simplicity.")
# %% [markdown]
# ---
#
# ## Key Takeaways
#
# Profile of the Binance premium-index panel underpinning the funding-arbitrage
# case study.
#
# ### Quantitative Findings
# - **Panel scale**: 107,839 8h observations across 19 USDT-margined perpetuals,
# 2020-01-01 → 2025-12-31. Coverage ranges from BTC/ETH (6,555 obs) down to
# SUIUSDT (2,920 obs from May 2023).
# - **Slight short bias**: All 19 symbols have a *negative* mean premium
# (between 0.03 and 2.6 bps; MKR is essentially flat), so on average
# perpetuals trade *below* spot — the raw funding flow is from shorts to
# longs before adding the interest-rate baseline.
# - **Volatility spectrum**: BTC has the tightest premium (std 5.6 bps).
# ETH/ADA/DOT cluster at 67 bps (~1.2× BTC). The wide-tail altcoins are
# SOL (std 36.7 bps, min 1,915 bps during the FTX collapse), XRP/UNI/COMP
# (1011 bps), reflecting episodic dislocation rather than steady-state
# volatility.
# - **Arbitrage frequency**: |premium| > 10 bps in 514 % of 8h periods
# depending on the symbol (BTC 5.0 %, COMP/ATOM 1314 %).
# - **Funding APY**: Binance's clamped funding rate (±0.05 % + 0.01 % interest)
# bounds the BTC annualized return at +65.7 % / 43.8 %. Realised mean is
# 5.7 % over 2020-25; the clamp is hit in 82.9 % of 8h periods (driven by
# the interest-rate baseline pushing |APY| above 20 % whenever premium is
# small).
#
# ### Implications for the Funding-Arbitrage Case Study
# - **Direction matters**: The negative mean premium means a *delta-neutral
# short-spot / long-perpetual* leg captures the structural funding flow on
# average for these symbols; the mirror trade only profits during transient
# bullish dislocations.
# - **Asset selection**: Wide-tail altcoins (SOL, XRP, COMP) offer the largest
# per-event funding but expose the strategy to extreme premium tails.
# BTC/ETH provide a tighter, more reliable funding stream.
# - **Regime awareness**: The 30-day rolling premium swings between bull (2021)
# and bear (2022) regimes; static thresholds will mis-fire — see the
# `case_studies/crypto_perps_funding/` pipeline for the regime-aware signal
# used downstream.
#
# **Next**: `12_fx_pairs_eda` profiles the third 24/7-adjacent dataset —
# G10 FX pairs at 4h cadence — completing the global market-data tour.