# ---
# jupyter:
#   jupytext:
#     cell_metadata_filter: tags,-all
#     text_representation:
#       extension: .py
#       format_name: percent
#       format_version: '1.3'
#       jupytext_version: 1.19.3
#   kernelspec:
#     display_name: Python 3 (ipykernel)
#     language: python
#     name: python3
# ---

# %% [markdown]
# # S&P 500 Options Analytics
#
# **Docker image**: `ml4t`
#
# ## Purpose
#
# Profile the AlgoSeek S&P 500 options analytics dataset for an 8-symbol 2020 EDA
# slice — chain structure, implied-volatility surfaces, data quality, and an early
# look at the predictive content that motivates the options case studies.
#
# ## Learning Objectives
#
# - Read an option chain and locate strikes, expirations, and call/put pairs.
# - Construct and visualize an implied-volatility smile, term structure, and surface.
# - Apply IV-convergence and Greeks-validity filters to clean options data.
# - Quantify a baseline IV-change → forward-return relationship in the cross section.
#
# ## Book Reference
#
# Chapter 2 §2.2 (asset-class market data landscape — derivatives).
#
# ## Prerequisites
#
# - Familiarity with daily OHLC equity data (`01_us_equities_eda`).
# - The AlgoSeek S&P 500 options EDA parquet at `$ML4T_DATA_PATH/sp500_options/`
#   (8 representative underlyings: AAPL, AMZN, BA, GOOGL, JPM, KO, MSFT, XOM).
# - The S&P 500 daily-bar parquet covering the same 2020 window.
#
# Loaders used:
#
# | Dataset | Loader | Coverage |
# |---------|--------|----------|
# | S&P 500 options (EDA slice) | `load_sp500_options_eda()` | 2020, 8 underlyings |
# | S&P 500 daily prices | `load_sp500_daily_bars()` | 2020, same 8 underlyings |

# %%
"""S&P 500 Options Analytics — options chain structure, volatility surfaces, and data quality."""

import plotly.express as px
import plotly.graph_objects as go
import polars as pl
from plotly.subplots import make_subplots

from data import load_sp500_daily_bars, load_sp500_options_eda

# %% tags=["parameters"]
DAILY_START_DATE = "2020-01-01"

# %% [markdown]
# ## 1. Options Primer for ML Practitioners
#
# Before diving into the data, let's establish the key concepts that make options
# data different from—and complementary to—spot market data.
#
# ### 1.1 What is an Option?
#
# An option is a **derivative contract** that gives the holder the right (but not
# obligation) to buy or sell an underlying asset at a specified price (strike)
# by a specified date (expiration).
#
# | Type | Right | Profitable When |
# |------|-------|-----------------|
# | **Call** | Buy at strike | Underlying rises above strike |
# | **Put** | Sell at strike | Underlying falls below strike |
#
# ### 1.2 Why Options Data Matters for ML
#
# Options prices embed **forward-looking information** that spot prices don't:
#
# 1. **Implied Volatility (IV)** - Market's expectation of future volatility
# 2. **IV Skew** - Relative demand for downside vs upside protection
# 3. **Term Structure** - How expectations change across time horizons
# 4. **Greeks** - Sensitivities that quantify risk exposures
#
# This information can predict:
# - Future realized volatility
# - Underlying price movements (via order flow/positioning)
# - Tail risk events (via skew)
#
# ### 1.3 Moneyness: ITM, ATM, OTM
#
# Moneyness describes how an option's strike relates to the current spot price:
#
# | Moneyness | Call (Strike vs Spot) | Put (Strike vs Spot) | Characteristics |
# |-----------|----------------------|---------------------|-----------------|
# | **ITM** (In-the-money) | Strike < Spot | Strike > Spot | Has intrinsic value |
# | **ATM** (At-the-money) | Strike ≈ Spot | Strike ≈ Spot | Highest time value |
# | **OTM** (Out-of-the-money) | Strike > Spot | Strike < Spot | Pure time value |
#
# We typically express moneyness as: **Strike / Spot** (or its log)
# - Moneyness = 1.0 → ATM
# - Moneyness < 1.0 → ITM call / OTM put
# - Moneyness > 1.0 → OTM call / ITM put
#
# ### 1.4 Option Value Components
#
# An option's price decomposes into intrinsic value (immediate exercise payoff) and
# time value (the remainder, reflecting optionality):
#
# $$\text{Price} = \text{Intrinsic} + \text{Time}$$
#
# $$\text{Intrinsic}_{\text{call}} = \max(0,\; S - K), \qquad \text{Intrinsic}_{\text{put}} = \max(0,\; K - S)$$
#
# $$\text{Time} = \text{Price} - \text{Intrinsic}$$
#
# Time value reflects:
# - Time remaining until expiration
# - Expected volatility (IV)
# - Interest rates and dividends

# %% [markdown]
# ## 2. Dataset Overview
#
# ### 2.1 Data Schema
#
# | Field | Type | Description |
# |-------|------|-------------|
# | **Identifiers** | | |
# | `timestamp` | Date | Trading date (observation date, EOD snapshot) |
# | `symbol` | String | Underlying ticker (e.g., "AAPL", "MSFT") |
# | `expiration` | Date | Option expiration date |
# | `strike` | Float64 | Strike price in USD |
# | `call_put` | String | "C" for call, "P" for put |
# | **Prices** | | |
# | `bid` | Float64 | Best bid price at close |
# | `ask` | Float64 | Best ask price at close |
# | `mid_price` | Float64 | Mid-market price: (bid + ask) / 2 |
# | `underlying_price` | Float64 | Underlying stock close price |
# | **Time** | | |
# | `days_to_maturity` | Int32 | Calendar days until expiration |
# | **Greeks** | | |
# | `delta` | Float64 | ∂V/∂S - Price sensitivity to underlying |
# | `gamma` | Float64 | ∂²V/∂S² - Delta sensitivity to underlying |
# | `theta` | Float64 | ∂V/∂t - Time decay ($/day, typically negative) |
# | `vega` | Float64 | ∂V/∂σ - Sensitivity to volatility |
# | `rho` | Float64 | ∂V/∂r - Sensitivity to interest rates |
# | **Volatility** | | |
# | `implied_vol` | Float64 | Black-Scholes implied volatility |
# | `iv_convergence` | String | IV solver status (quality indicator) |
#
# ### 2.2 IV Convergence Codes
#
# The `iv_convergence` field indicates IV computation quality:
#
# | Code | Meaning | Use in Analysis |
# |------|---------|-----------------|
# | `Converged` | IV solver converged normally | [OK] Highest quality |
# | `SmallBid_FlatExtrapol` | Small bid, IV extrapolated | WARNING: Use with caution |
# | `IntrVal_FlatExtrapol` | Deep ITM, IV extrapolated | WARNING: Use with caution |
# | `IntrVal_PutCallPair` | IV from put-call parity | [OK] Usually reliable |
# | `Failed` | IV solver did not converge | [FAIL] Exclude from analysis |
#
# **Best practice**: Filter for `iv_convergence == "Converged"` for clean analysis.

# %%
options = load_sp500_options_eda(
    start_date="2020-01-01",
    end_date="2020-12-31",
    include_greeks=True,
)

print("=== S&P 500 Options Dataset ===")
print(f"Total rows: {len(options):,}")
print(f"Columns: {len(options.columns)}")
print(f"Date range: {options['timestamp'].min()} to {options['timestamp'].max()}")
print(f"Underlyings: {sorted(options['symbol'].unique().to_list())}")

# %%
daily = load_sp500_daily_bars(
    symbols=sorted(options["symbol"].unique().to_list()),
    start_date=DAILY_START_DATE,
    end_date="2020-12-31",
)

print("\n=== S&P 500 Daily Prices ===")
print(f"Total rows: {len(daily):,}")
print(f"Symbols: {daily['symbol'].n_unique()}")
print(f"Date range: {daily['timestamp'].min()} to {daily['timestamp'].max()}")

# %%
# Quick schema preview
print("\n=== Options Schema ===")
options.head(3)

# %% [markdown]
# ## 3. Option Chain Structure
#
# An **option chain** is the full set of options available for one underlying on one day.
# It spans multiple dimensions:
# - **Strikes**: Many price levels around the current spot
# - **Expirations**: Multiple dates from days to years out
# - **Types**: Calls and puts at each strike/expiration
#
# This creates a 3D grid: `(strike × expiration × call_put)`

# %%
# Options per symbol per day - how dense are the chains?
options_per_symbol = options.group_by(["timestamp", "symbol"]).agg(
    [
        pl.len().alias("n_options"),
        (pl.col("call_put") == "C").sum().alias("n_calls"),
        (pl.col("call_put") == "P").sum().alias("n_puts"),
        pl.col("strike").n_unique().alias("n_strikes"),
        pl.col("expiration").n_unique().alias("n_expirations"),
    ]
)

print("=== Option Chain Density (per symbol per day) ===")
options_per_symbol.select(["n_options", "n_strikes", "n_expirations"]).describe()

# %%
# Visualize: Distribution of chain sizes
fig = px.histogram(
    options_per_symbol.to_pandas(),
    x="n_options",
    nbins=50,
    title="Option Chain Size Distribution",
    labels={"n_options": "Number of Options per Symbol/Day", "count": "Frequency"},
)
median_options = float(options_per_symbol["n_options"].median())
fig.add_vline(x=median_options, line_dash="dash", line_color="red")
fig.add_annotation(x=median_options, y=0.95, yref="paper", text="Median", showarrow=False)
fig.update_layout(showlegend=False)
fig.show()

# %% [markdown]
# ### 3.1 Single Symbol Deep Dive: AAPL
#
# Let's examine one complete option chain to understand the structure.

# %%
# AAPL on a specific date
sample_date = options["timestamp"].max()
aapl_day = options.filter((pl.col("symbol") == "AAPL") & (pl.col("timestamp") == sample_date))

spot = aapl_day["underlying_price"][0]

print(f"=== AAPL Option Chain ({sample_date}) ===")
print(f"Underlying price: ${spot:.2f}")
print(f"Total options: {len(aapl_day):,}")
print(f"  Calls: {aapl_day.filter(pl.col('call_put') == 'C').height:,}")
print(f"  Puts: {aapl_day.filter(pl.col('call_put') == 'P').height:,}")
print(f"Expirations: {aapl_day['expiration'].n_unique()}")
print(f"Strikes: {aapl_day['strike'].n_unique()}")
print(f"Strike range: ${aapl_day['strike'].min():.2f} - ${aapl_day['strike'].max():.2f}")

# %%
# Expiration breakdown
exp_breakdown = (
    aapl_day.group_by("expiration")
    .agg([pl.len().alias("n_options"), pl.col("strike").n_unique().alias("n_strikes")])
    .sort("expiration")
)
print("\n=== AAPL Expirations ===")
exp_breakdown.head(10)

# %% [markdown]
# ### 3.2 Option Chain Heatmap
#
# Visualize the entire chain as a heatmap: strikes on y-axis, expirations on x-axis,
# colored by implied volatility. This reveals the **volatility surface** structure.

# %%
# Prepare data for heatmap - calls only, converged IV, reasonable moneyness
aapl_calls = (
    aapl_day.filter(
        (pl.col("call_put") == "C")
        & (pl.col("iv_convergence") == "Converged")
        & (pl.col("implied_vol") > 0)
        & (pl.col("implied_vol") < 2.0)  # Filter outliers
    )
    .with_columns((pl.col("strike") / pl.col("underlying_price")).alias("moneyness"))
    .filter(pl.col("moneyness").is_between(0.7, 1.3))  # Focus on tradeable range
)

# Create pivot for heatmap
heatmap_data = (
    aapl_calls.select(["strike", "expiration", "implied_vol"])
    .sort(["expiration", "strike"])
    .to_pandas()
    .pivot(index="strike", columns="expiration", values="implied_vol")
)

# %%
fig = go.Figure(
    data=go.Heatmap(
        z=heatmap_data.values,
        x=[str(c) for c in heatmap_data.columns],
        y=heatmap_data.index,
        colorscale="Viridis",
        colorbar=dict(title="IV"),
    )
)

spot_float = float(spot)
fig.add_hline(y=spot_float, line_dash="dash", line_color="white")
fig.add_annotation(
    y=spot_float, x=0.95, xref="paper", text=f"Spot: ${spot_float:.0f}", showarrow=False
)

fig.update_layout(
    title=f"AAPL Option Chain - Implied Volatility Surface ({sample_date})",
    xaxis_title="Expiration",
    yaxis_title="Strike ($)",
    height=600,
)
fig.show()

# %% [markdown]
# **Reading the heatmap:**
# - Horizontal slice at one strike → Term structure (how IV varies with expiration)
# - Vertical slice at one expiration → IV smile/skew (how IV varies with strike)
# - Darker colors (lower IV) typically at ATM; lighter (higher IV) at wings

# %% [markdown]
# ## 4. Volatility Surface Analysis
#
# The **implied volatility surface** is the core representation for options analytics.
# It captures how IV varies across two dimensions:
# 1. **Moneyness** (strike relative to spot) → IV smile/skew
# 2. **Time to expiration** → IV term structure
#
# ### 4.1 IV Smile and Skew

# %%
# IV smile for nearest expiration
nearest_exp = aapl_calls["expiration"].min()
aapl_smile = aapl_calls.filter(pl.col("expiration") == nearest_exp).sort("strike")

fig = px.scatter(
    aapl_smile.to_pandas(),
    x="moneyness",
    y="implied_vol",
    title=f"AAPL IV Smile - Nearest Expiration ({nearest_exp})",
    labels={"moneyness": "Moneyness (Strike/Spot)", "implied_vol": "Implied Volatility"},
    trendline="lowess",
)
fig.add_vline(x=1.0, line_dash="dash", line_color="gray")
fig.add_annotation(x=1.0, y=0.95, yref="paper", text="ATM", showarrow=False)
fig.show()

# %% [markdown]
# **IV Smile/Skew interpretation:**
# - **Smile**: IV higher for both OTM puts (left) and OTM calls (right) vs ATM
# - **Skew**: Asymmetric - OTM puts typically have higher IV than OTM calls
# - **Why?** Demand for downside protection (crash insurance) exceeds upside speculation

# %%
# Compare smile across multiple expirations
expirations = sorted(aapl_calls["expiration"].unique().to_list())[:4]  # First 4

smile_data = aapl_calls.filter(pl.col("expiration").is_in(expirations))

fig = px.scatter(
    smile_data.to_pandas(),
    x="moneyness",
    y="implied_vol",
    color="expiration",
    title="AAPL IV Smile Across Expirations",
    labels={"moneyness": "Moneyness", "implied_vol": "Implied Volatility"},
)
fig.add_vline(x=1.0, line_dash="dash", line_color="gray")
fig.show()

# %% [markdown]
# ### 4.2 IV Term Structure
#
# How does ATM IV vary across expirations?

# %%
# ATM IV term structure (moneyness 0.98-1.02)
atm_term = (
    aapl_calls.filter(pl.col("moneyness").is_between(0.98, 1.02))
    .group_by("expiration")
    .agg(
        [
            pl.col("implied_vol").mean().alias("iv_atm"),
            pl.col("days_to_maturity").first().alias("days"),
        ]
    )
    .sort("expiration")
)

fig = px.line(
    atm_term.to_pandas(),
    x="days",
    y="iv_atm",
    markers=True,
    title=f"AAPL ATM IV Term Structure ({sample_date})",
    labels={"days": "Days to Expiration", "iv_atm": "ATM Implied Volatility"},
)
fig.show()

# %% [markdown]
# **Term structure shapes:**
# - **Contango** (upward sloping): Near-term calm, uncertainty further out
# - **Backwardation** (downward sloping): Near-term stress/event expected
# - **Flat**: Consistent expectations across horizons

# %% [markdown]
# ### 4.3 3D Volatility Surface
#
# Combine moneyness and time dimensions into a single surface visualization.

# %%
# Prepare surface data
surface_data = (
    aapl_calls.filter(pl.col("days_to_maturity") <= 180)  # Focus on <6 months
    .select(["moneyness", "days_to_maturity", "implied_vol"])
    .to_pandas()
)

# Create 3D surface
fig = go.Figure(
    data=[
        go.Mesh3d(
            x=surface_data["moneyness"],
            y=surface_data["days_to_maturity"],
            z=surface_data["implied_vol"],
            intensity=surface_data["implied_vol"],
            colorscale="Viridis",
            opacity=0.7,
        )
    ]
)

fig.update_layout(
    title=f"AAPL 3D Volatility Surface ({sample_date})",
    scene=dict(
        xaxis_title="Moneyness",
        yaxis_title="Days to Expiration",
        zaxis_title="Implied Volatility",
    ),
    height=600,
)
fig.show()

# %% [markdown]
# ## 5. Cross-Sectional Analysis
#
# How does ATM IV differ across the eight underlyings on a single day? The same
# logic scales to the full S&P 500 universe — here we keep the comparison
# tractable on the EDA slice.

# %%
# Compute ATM IV for all symbols on sample date
converged = options.filter(pl.col("iv_convergence") == "Converged")

cross_section = (
    converged.filter(pl.col("timestamp") == sample_date)
    .with_columns((pl.col("strike") / pl.col("underlying_price")).alias("moneyness"))
    .filter(pl.col("moneyness").is_between(0.98, 1.02))
    .filter(pl.col("call_put") == "C")
    .group_by("symbol")
    .agg(
        [
            pl.col("implied_vol").mean().alias("iv_atm"),
            pl.col("underlying_price").first().alias("price"),
        ]
    )
    .sort("iv_atm", descending=True)
)

print(f"=== Cross-Sectional ATM IV ({sample_date}) ===")
print(f"Symbols: {len(cross_section)}")
print(f"IV range: {cross_section['iv_atm'].min():.1%} - {cross_section['iv_atm'].max():.1%}")
print(f"IV median: {cross_section['iv_atm'].median():.1%}")

# %%
# Highest IV names
print("\n=== Highest IV Names ===")
cross_section.head(10)

# %%
# Lowest IV names
print("\n=== Lowest IV Names ===")
cross_section.tail(10)

# %%
# IV distribution across universe
fig = px.histogram(
    cross_section.to_pandas(),
    x="iv_atm",
    nbins=40,
    title=f"Cross-Sectional ATM IV Distribution ({sample_date})",
    labels={"iv_atm": "ATM Implied Volatility", "count": "Number of Symbols"},
)
median_iv = float(cross_section["iv_atm"].median())
fig.add_vline(x=median_iv, line_dash="dash", line_color="red")
fig.add_annotation(x=median_iv, y=0.95, yref="paper", text="Median", showarrow=False)
fig.show()

# %% [markdown]
# ## 6. Time Series Analysis
#
# How does IV evolve over time? The year 2020 provides an excellent case study
# with the COVID crash in March.

# %%
# Daily aggregate IV statistics
daily_iv = (
    converged.with_columns((pl.col("strike") / pl.col("underlying_price")).alias("moneyness"))
    .filter(pl.col("moneyness").is_between(0.98, 1.02))
    .filter(pl.col("call_put") == "C")
    .group_by("timestamp")
    .agg(
        [
            pl.col("implied_vol").mean().alias("iv_mean"),
            pl.col("implied_vol").median().alias("iv_median"),
            pl.col("implied_vol").quantile(0.1).alias("iv_p10"),
            pl.col("implied_vol").quantile(0.9).alias("iv_p90"),
            pl.col("symbol").n_unique().alias("n_symbols"),
        ]
    )
    .sort("timestamp")
)

# %%
# Build the full IV-evolution figure in one cell — splitting the figure across
# two cells produced an intermediate render with no title or axis labels.
fig = go.Figure()

fig.add_trace(
    go.Scatter(
        x=daily_iv["timestamp"].to_list(),
        y=daily_iv["iv_p90"].to_list(),
        fill=None,
        mode="lines",
        line_color="lightblue",
        name="P90",
    )
)
fig.add_trace(
    go.Scatter(
        x=daily_iv["timestamp"].to_list(),
        y=daily_iv["iv_p10"].to_list(),
        fill="tonexty",
        mode="lines",
        line_color="lightblue",
        name="P10-P90 Range",
    )
)
fig.add_trace(
    go.Scatter(
        x=daily_iv["timestamp"].to_list(),
        y=daily_iv["iv_median"].to_list(),
        mode="lines",
        line_color="darkblue",
        line_width=2,
        name="Median IV",
    )
)

fig.add_vline(x="2020-03-16", line_dash="dash", line_color="red")
fig.add_vline(x="2020-03-23", line_dash="dash", line_color="green")
fig.add_annotation(x="2020-03-16", y=0.95, yref="paper", text="COVID Low", showarrow=False)
fig.add_annotation(x="2020-03-23", y=0.90, yref="paper", text="Market Bottom", showarrow=False)

fig.update_layout(
    title="S&P 500 Universe ATM IV Evolution (2020)",
    xaxis_title="Date",
    yaxis_title="Implied Volatility",
    height=500,
)
fig.show()

# %% [markdown]
# **Key observations:**
# - The 90th-percentile ATM IV crossed 100% in mid-March 2020 (peak P90 ≈ 115% on
#   2020-03-16); cross-sectional median ATM IV peaked near 70%.
# - IV stayed materially above pre-crash levels well past Q1 — slow mean reversion
#   even as spot recovered.
# - The "fear gauge" aspect of IV — sharp spike, slow decay — is visible even on
#   this eight-symbol slice.

# %% [markdown]
# ## 7. Execution Cost Proxy: Bid-Ask Spreads
#
# Since we don't have volume or open interest, bid-ask spread serves as our
# primary liquidity/execution cost indicator.

# %%
# Compute spread metrics
spread_analysis = (
    converged.with_columns(
        [
            (pl.col("ask") - pl.col("bid")).alias("spread_abs"),
            ((pl.col("ask") - pl.col("bid")) / pl.col("mid_price")).alias("spread_pct"),
            (pl.col("strike") / pl.col("underlying_price")).alias("moneyness"),
        ]
    )
    .filter(pl.col("mid_price") > 0.10)  # Filter penny options
    .filter(pl.col("spread_pct") < 2.0)  # Filter outliers
)

print("=== Bid-Ask Spread Statistics ===")
spread_analysis.select(["spread_abs", "spread_pct"]).describe()

# %%
# Spread by moneyness
spread_by_moneyness = (
    spread_analysis.filter(pl.col("moneyness").is_between(0.8, 1.2))
    .with_columns((pl.col("moneyness") * 20).round() / 20)  # Bucket to 5% increments
    .group_by("moneyness")
    .agg([pl.col("spread_pct").median().alias("median_spread")])
    .sort("moneyness")
)

fig = px.bar(
    spread_by_moneyness.to_pandas(),
    x="moneyness",
    y="median_spread",
    title="Median Bid-Ask Spread by Moneyness",
    labels={"moneyness": "Moneyness", "median_spread": "Median Spread (%)"},
)
fig.add_vline(x=1.0, line_dash="dash", line_color="gray")
fig.add_annotation(x=1.0, y=0.95, yref="paper", text="ATM", showarrow=False)
fig.show()

# %% [markdown]
# **Spread observations:**
# - ATM options have tightest spreads (most liquid)
# - Spreads widen for OTM options (less liquid)
# - Deep OTM options can have very wide spreads (>50%)
# - **Implication**: Focus on near-ATM for tradeable strategies

# %% [markdown]
# ## 8. Data Quality Assessment
#
# ### 8.1 IV Convergence Rates

# %%
# Convergence statistics
convergence_stats = (
    options.group_by("iv_convergence")
    .len()
    .with_columns((pl.col("len") / pl.sum("len") * 100).alias("pct"))
    .sort("len", descending=True)
)

print("=== IV Convergence Status ===")
for row in convergence_stats.iter_rows(named=True):
    print(f"  {row['iv_convergence']}: {row['len']:,} ({row['pct']:.2f}%)")

# %%
# Horizontal bar chart instead of a pie: 10 convergence categories make a pie
# unreadable (tiny slices overlap their labels). The bar chart sorts by share
# and keeps every label legible.
convergence_pd = convergence_stats.to_pandas().sort_values("pct", ascending=True)
fig = go.Figure(
    data=go.Bar(
        x=convergence_pd["pct"],
        y=convergence_pd["iv_convergence"],
        orientation="h",
        text=[f"{v:.2f}%" for v in convergence_pd["pct"]],
        textposition="outside",
        marker_color="#4C72B0",
    )
)
fig.update_layout(
    title="IV Convergence Status Distribution",
    xaxis_title="Share of rows (%)",
    yaxis_title="Convergence status",
    template="plotly_white",
    height=460,
    margin=dict(l=170, r=100),
    xaxis=dict(range=[0, max(convergence_pd["pct"]) * 1.15]),
)
fig.show()

# %% [markdown]
# ### 8.2 Coverage Analysis

# %%
# Daily symbol coverage
daily_coverage = (
    converged.group_by("timestamp")
    .agg(
        [
            pl.col("symbol").n_unique().alias("n_symbols"),
            pl.len().alias("n_options"),
        ]
    )
    .sort("timestamp")
)

print("=== Daily Coverage (Converged Options) ===")
print(f"Mean symbols/day: {daily_coverage['n_symbols'].mean():.0f}")
print(f"Min symbols/day: {daily_coverage['n_symbols'].min()}")
print(f"Max symbols/day: {daily_coverage['n_symbols'].max()}")

# %%
fig = make_subplots(
    rows=2,
    cols=1,
    shared_xaxes=True,
    subplot_titles=("Symbols with Converged Options", "Total Converged Options"),
)

coverage_pd = daily_coverage.to_pandas()

fig.add_trace(
    go.Scatter(x=coverage_pd["timestamp"], y=coverage_pd["n_symbols"], name="Symbols"),
    row=1,
    col=1,
)
fig.add_trace(
    go.Scatter(x=coverage_pd["timestamp"], y=coverage_pd["n_options"], name="Options"),
    row=2,
    col=1,
)

fig.update_layout(height=500, title="Options Universe Coverage Over Time")
fig.show()

# %% [markdown]
# ### 8.3 Greeks Validation

# %%
print("=== Greeks Validity Checks ===")

checks = converged.select(
    [
        # Delta should be [-1, 1]
        ((pl.col("delta") >= -1.0) & (pl.col("delta") <= 1.0)).mean().alias("delta_in_bounds"),
        # Gamma should be non-negative
        (pl.col("gamma") >= 0).mean().alias("gamma_non_negative"),
        # Vega should be non-negative
        (pl.col("vega") >= 0).mean().alias("vega_non_negative"),
        # Theta typically negative
        (pl.col("theta") <= 0.01).mean().alias("theta_typical"),
        # IV should be positive
        (pl.col("implied_vol") > 0).mean().alias("iv_positive"),
    ]
)

for col in checks.columns:
    pct = checks[col][0] * 100
    status = "PASS" if pct > 99.9 else ("WARN" if pct > 95 else "FAIL")
    print(f"  [{status}] {col}: {pct:.2f}%")

# %%
print("\n=== Greeks Summary Statistics ===")
converged.select(["delta", "gamma", "theta", "vega", "implied_vol"]).describe()

# %% [markdown]
# ### 8.4 Point-in-Time Validation

# %%
print("=== Point-in-Time Checks ===")

# Expiration must be >= observation date
exp_check = options.filter(pl.col("expiration") < pl.col("timestamp"))
print(f"Expiration < Date violations: {len(exp_check):,}")

# Days to maturity must be non-negative
dtm_check = options.filter(pl.col("days_to_maturity") < 0)
print(f"Negative days_to_maturity: {len(dtm_check):,}")

if len(exp_check) == 0 and len(dtm_check) == 0:
    print("[OK] PASSED - No look-ahead bias detected")
else:
    print("[FAIL] FAILED - Data integrity issue")

# %% [markdown]
# ## 9. Information Content Preview
#
# Why does options data predict underlying returns? Let's examine the IV-return
# relationship.

# %%
# Compute ATM IV per symbol/date
atm_iv = (
    converged.with_columns((pl.col("strike") / pl.col("underlying_price")).alias("moneyness"))
    .filter(pl.col("moneyness").is_between(0.98, 1.02))
    .filter(pl.col("call_put") == "C")
    .with_columns((pl.col("moneyness") - 1.0).abs().alias("atm_distance"))
    .sort(["timestamp", "symbol", "atm_distance"])
    .group_by(["timestamp", "symbol"])
    .first()
    .select(["timestamp", "symbol", "implied_vol", "underlying_price"])
    .rename({"implied_vol": "iv_atm"})
)

# Join with daily prices and compute returns
panel = (
    atm_iv.join(
        daily.select(["timestamp", "symbol", "close"]), on=["timestamp", "symbol"], how="inner"
    )
    .sort(["symbol", "timestamp"])
    .with_columns(
        [
            pl.col("iv_atm").shift(5).over("symbol").alias("iv_atm_lag5"),
            pl.col("close").shift(-5).over("symbol").alias("close_fwd5"),
        ]
    )
    .with_columns(
        [
            (pl.col("iv_atm") - pl.col("iv_atm_lag5")).alias("iv_change_5d"),
            ((pl.col("close_fwd5") / pl.col("close")) - 1).alias("ret_fwd5"),
        ]
    )
    .drop_nulls(subset=["iv_change_5d", "ret_fwd5"])
)

# Correlation
correlation = panel.select(pl.corr("iv_change_5d", "ret_fwd5").alias("corr"))[0, 0]
print("=== IV Change vs Forward Return ===")
print(f"Correlation: {correlation:.4f}")
print("Interpretation: Falling IV tends to precede positive returns")

# %%
# Quintile analysis
panel_ranked = (
    panel.with_columns(pl.col("iv_change_5d").rank().over("timestamp").alias("iv_rank_raw"))
    .with_columns(
        (pl.col("iv_rank_raw") / pl.col("iv_rank_raw").max().over("timestamp") * 100).alias(
            "iv_pct"
        )
    )
    .with_columns(
        pl.when(pl.col("iv_pct") <= 20)
        .then(1)
        .when(pl.col("iv_pct") <= 40)
        .then(2)
        .when(pl.col("iv_pct") <= 60)
        .then(3)
        .when(pl.col("iv_pct") <= 80)
        .then(4)
        .otherwise(5)
        .alias("iv_quintile")
    )
)

quintile_returns = (
    panel_ranked.group_by("iv_quintile")
    .agg(
        [
            pl.col("ret_fwd5").mean().alias("mean_ret"),
            pl.col("ret_fwd5").std().alias("std_ret"),
            pl.len().alias("n_obs"),
        ]
    )
    .sort("iv_quintile")
)

print("\n=== Forward Returns by IV Change Quintile ===")
print("(Q1 = largest IV decrease, Q5 = largest IV increase)")
quintile_returns

# %%
fig = px.bar(
    quintile_returns.to_pandas(),
    x="iv_quintile",
    y="mean_ret",
    title="5-Day Forward Returns by IV Change Quintile",
    labels={
        "iv_quintile": "IV Change Quintile (1=falling, 5=rising)",
        "mean_ret": "Mean 5-Day Return",
    },
)
fig.update_layout(xaxis=dict(tickmode="array", tickvals=[1, 2, 3, 4, 5]))
fig.show()

# %% [markdown]
# The sign of the −0.08 IV-change vs forward-return correlation is negative: on
# this 8-symbol 2020 slice, larger IV declines line up with higher 5-day forward
# returns and larger IV increases with lower forward returns. The quintile means
# above show how monotonic the relationship is. This notebook does not test
# statistical significance or out-of-sample stability; the IV-based feature is
# evaluated rigorously via IC analysis in Chapter 9.

# %% [markdown]
# ## 10. Data Quality Summary

# %%
total_rows = len(options)
converged_rows = len(converged)
converged_pct = converged_rows / total_rows * 100

print("=" * 70)
print("DATA QUALITY SUMMARY: S&P 500 OPTIONS")
print("=" * 70)

print("\n1. SCALE")
print(f"   Total options records: {total_rows:,}")
print(f"   Converged IV records: {converged_rows:,} ({converged_pct:.1f}%)")
print(f"   Unique underlyings: {options['symbol'].n_unique()}")
print(f"   Date range: {options['timestamp'].min()} to {options['timestamp'].max()}")

print("\n2. COVERAGE")
print(f"   Trading days: {daily_coverage['timestamp'].n_unique()}")
print(f"   Avg symbols/day: {daily_coverage['n_symbols'].mean():.0f}")
print(f"   Avg options/symbol/day: {options_per_symbol['n_options'].mean():.0f}")

print("\n3. DATA QUALITY")
print(f"   Point-in-time: {'PASS' if len(exp_check) == 0 else 'FAIL'}")
print("   Greeks validity: See checks above")
print(f"   Convergence rate: {converged_pct:.1f}%")

print("\n4. EXECUTION PROXY")
print(
    f"   Median ATM spread: {spread_by_moneyness.filter(pl.col('moneyness') == 1.0)['median_spread'][0]:.1%}"
)

print("\n5. INFORMATION CONTENT")
print(f"   IV-Return Correlation: {correlation:.4f}")

print("\n" + "=" * 70)
print("DATASET SUPPORTS TWO CASE STUDIES:")
print("  sp500_equity_option_analytics — IV features used to trade equities")
print("  sp500_options                  — short-straddle harvest with daily delta hedge")
print("=" * 70)

# %% [markdown]
# ## Key Takeaways
#
# 1. **Scale (EDA slice)**: ~5.2M option records across 8 representative S&P 500
#    underlyings in 2020. The full AlgoSeek panel covers the broader index — this
#    notebook intentionally subsamples for fast exploration.
# 2. **Structure**: Each underlying carries dozens of expirations and 100+
#    strikes per day; chains are 3-D grids (strike × expiration × call/put).
# 3. **Quality**: 69.6% of rows carry `iv_convergence == "Converged"`. The other
#    convergence codes flag extrapolated or solver-failed IVs and should be
#    excluded from analysis.
# 4. **Greeks**: Pre-computed Black-Scholes sensitivities are well-bounded — Δ in
#    [-1, 1], Γ ≥ 0, ν ≥ 0, θ ≤ 0 — at >99.9% of converged rows.
# 5. **Volatility Surface**: Smile, skew, and term structure are all visible on
#    this slice; the 3-D surface compresses both axes into one plot.
# 6. **Information**: 5-day IV changes correlate with 5-day forward equity
#    returns at -0.08 on this sample (falling IV → positive returns), motivating
#    the IV-based features in Chapter 8.
# 7. **Execution**: Median ATM bid-ask spread is ~6%; deep-OTM spreads widen
#    sharply, which constrains tradeable strategies to near-the-money strikes.
#
# ## Data Limitations
#
# - **No volume/open interest**: Cannot filter for liquidity directly.
# - **Daily snapshots only**: No intraday dynamics for gamma scalping.
# - **Spread as proxy**: Bid-ask is the only execution cost indicator available.
#
# ## Next Steps
#
# - `08_options_greeks_computation`: Compute Greeks from scratch and validate
#   against the vendor numbers used here.
# - Chapter 8: Build IV surface features for ML.
# - Chapter 9: Model-based feature extraction (PCA, autoencoders).
# - Chapter 12: ML models for equity and options targets.
# - Chapter 16: Backtests using these features.