{ "cells": [ { "cell_type": "markdown", "id": "9bd10cbb", "metadata": { "papermill": { "duration": 0.001947, "end_time": "2026-06-13T02:50:55.958572+00:00", "exception": false, "start_time": "2026-06-13T02:50:55.956625+00:00", "status": "completed" } }, "source": [ "# Text Feature Signal Evaluation\n", "\n", "**Chapter 10: Text Feature Engineering**\n", "**Section Reference**: See Section 10.5 for practitioner workflow and alpha factor design\n", "\n", "**Docker image**: `ml4t`\n", "\n", "## Purpose\n", "This notebook evaluates **text-derived alpha signals** produced by\n", "[`07_news_return_signals`](07_news_return_signals.ipynb) using standard factor diagnostics:\n", "- Daily cross-sectional Information Coefficient (Spearman rank correlation)\n", "- ICIR and t-stats for robustness\n", "- Quintile and long-short (Q5 - Q1) spread analysis\n", "\n", "## Data Contract\n", "- **Input**: `output/fnspid/news_features.parquet` (from notebook 07)\n", " - Contains forward returns (fwd_ret_1d, fwd_ret_5d, fwd_ret_20d) already aligned\n", " - No additional price data or label computation needed\n", "- **Output**: `output/text_evaluation/text_signal_summary.parquet`\n", "\n", "## Learning Objectives\n", "After completing this notebook, you will be able to:\n", "- Evaluate NLP-derived alpha signals using standard factor analysis\n", "- Compute daily IC, ICIR, and t-statistics for text signals\n", "- Analyze quintile spreads and long-short returns\n", "- Understand the predictive power of text-based factors\n", "\n", "## Prerequisites\n", "- Section 10.5 of the chapter (alpha-factor evaluation: IC, ICIR, quintile spread).\n", "- `news_features.parquet` produced by `07_news_return_signals.py`.\n", "\n", "## Related Notebooks\n", "- `07_news_return_signals.py` — produces the text features evaluated here.\n", "- `09_filing_text_signals.py` — analogous IC analysis for 10-Q filing signals.\n", "\n", "## Signals Evaluated\n", "| Signal | Description | Expected Effect |\n", "|--------|-------------|-----------------|\n", "| weighted_surprise | News surprise × sentiment direction | Positive (directional information) |\n", "| sentiment_mean | Average daily sentiment | Positive (bullish → positive returns) |\n", "| sentiment_momentum | Change in sentiment vs baseline | Positive (improving sentiment) |\n", "| coverage_count | Article frequency | Ambiguous (attention effect) |" ] }, { "cell_type": "code", "execution_count": 1, "id": "642b602f", "metadata": { "execution": { "iopub.execute_input": "2026-06-13T02:50:55.962309Z", "iopub.status.busy": "2026-06-13T02:50:55.962143Z", "iopub.status.idle": "2026-06-13T02:50:58.230451Z", "shell.execute_reply": "2026-06-13T02:50:58.229842Z" }, "papermill": { "duration": 2.270841, "end_time": "2026-06-13T02:50:58.230946+00:00", "exception": false, "start_time": "2026-06-13T02:50:55.960105+00:00", "status": "completed" } }, "outputs": [], "source": [ "\"\"\"Text Feature Signal Evaluation - Evaluate NLP-derived alpha signals.\"\"\"\n", "\n", "import json\n", "import warnings\n", "from collections.abc import Iterator\n", "from dataclasses import dataclass\n", "\n", "import matplotlib.pyplot as plt\n", "import numpy as np\n", "import polars as pl\n", "from scipy.stats import spearmanr\n", "\n", "warnings.filterwarnings(\"ignore\")\n", "\n", "from utils.paths import get_output_dir\n", "from utils.reproducibility import set_global_seeds" ] }, { "cell_type": "code", "execution_count": 2, "id": "4b89cfa8", "metadata": { "execution": { "iopub.execute_input": "2026-06-13T02:50:58.234771Z", "iopub.status.busy": "2026-06-13T02:50:58.234674Z", "iopub.status.idle": "2026-06-13T02:50:58.236495Z", "shell.execute_reply": "2026-06-13T02:50:58.236122Z" }, "papermill": { "duration": 0.00444, "end_time": "2026-06-13T02:50:58.236778+00:00", "exception": false, "start_time": "2026-06-13T02:50:58.232338+00:00", "status": "completed" }, "tags": [ "parameters" ] }, "outputs": [], "source": [ "SEED = 42\n", "MIN_ASSETS_PER_DAY = 5" ] }, { "cell_type": "code", "execution_count": 3, "id": "003e3518", "metadata": { "execution": { "iopub.execute_input": "2026-06-13T02:50:58.239572Z", "iopub.status.busy": "2026-06-13T02:50:58.239498Z", "iopub.status.idle": "2026-06-13T02:50:58.279249Z", "shell.execute_reply": "2026-06-13T02:50:58.278709Z" }, "papermill": { "duration": 0.042189, "end_time": "2026-06-13T02:50:58.280167+00:00", "exception": false, "start_time": "2026-06-13T02:50:58.237978+00:00", "status": "completed" } }, "outputs": [], "source": [ "# Reproducibility — set_global_seeds covers Python random / NumPy / Torch.\n", "set_global_seeds(SEED)\n", "\n", "\n", "@dataclass(frozen=True)\n", "class EvalConfig:\n", " \"\"\"Configuration for text signal evaluation.\"\"\"\n", "\n", " horizons: tuple[int, ...] = (1, 5, 20)\n", " min_assets_per_day: int = 20\n", " quantiles: int = 5\n", "\n", "\n", "# Paths using standard utilities\n", "TEXT_INPUT_DIR = get_output_dir(8, \"fnspid\")\n", "OUTPUT_DIR = get_output_dir(8, \"text_evaluation\")\n", "CONFIG = EvalConfig(min_assets_per_day=MIN_ASSETS_PER_DAY)" ] }, { "cell_type": "markdown", "id": "81d673d5", "metadata": { "papermill": { "duration": 0.001164, "end_time": "2026-06-13T02:50:58.282697+00:00", "exception": false, "start_time": "2026-06-13T02:50:58.281533+00:00", "status": "completed" } }, "source": [ "## 1. Load Text Features from Notebook 07\n", "\n", "Notebook 07 produces a complete dataset with:\n", "- Text signals (weighted_surprise, sentiment_mean, etc.)\n", "- Forward returns (fwd_ret_1d, fwd_ret_5d, fwd_ret_20d) already computed\n", "- Proper look-ahead bias handling (signals lag returns by 1 day)\n", "\n", "We do NOT recompute labels here—we use the dataset contract from notebook 07." ] }, { "cell_type": "code", "execution_count": 4, "id": "00e93739", "metadata": { "execution": { "iopub.execute_input": "2026-06-13T02:50:58.285709Z", "iopub.status.busy": "2026-06-13T02:50:58.285583Z", "iopub.status.idle": "2026-06-13T02:50:58.295188Z", "shell.execute_reply": "2026-06-13T02:50:58.294782Z" }, "lines_to_next_cell": 2, "papermill": { "duration": 0.011594, "end_time": "2026-06-13T02:50:58.295446+00:00", "exception": false, "start_time": "2026-06-13T02:50:58.283852+00:00", "status": "completed" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Loaded 15,778 observations\n", "Columns: ['ticker', 'date', 'news_date', 'trade_date', 'news_surprise', 'sentiment_mean', 'sentiment_std', 'weighted_surprise', 'sentiment_momentum', 'coverage_count', 'fwd_ret_1d', 'fwd_ret_5d', 'fwd_ret_20d']\n" ] } ], "source": [ "# Load text-derived features\n", "text_features_path = TEXT_INPUT_DIR / \"news_features.parquet\"\n", "\n", "if not text_features_path.exists():\n", " raise FileNotFoundError(\n", " f\"Missing input: {text_features_path}\\n\\n\"\n", " \"Run notebook 07 first to generate this dataset:\\n\"\n", " \" python 08_text_feature_engineering/code/07_news_return_signals.py\"\n", " )\n", "\n", "text_features = pl.read_parquet(text_features_path)\n", "\n", "if len(text_features) == 0:\n", " raise ValueError(\n", " f\"Empty dataset: {text_features_path}\\nNotebook 07 ran but produced no usable rows.\"\n", " )\n", "\n", "print(f\"Loaded {len(text_features):,} observations\")\n", "print(f\"Columns: {text_features.columns}\")" ] }, { "cell_type": "code", "execution_count": 5, "id": "564c6def", "metadata": { "execution": { "iopub.execute_input": "2026-06-13T02:50:58.298394Z", "iopub.status.busy": "2026-06-13T02:50:58.298325Z", "iopub.status.idle": "2026-06-13T02:50:58.303730Z", "shell.execute_reply": "2026-06-13T02:50:58.303433Z" }, "papermill": { "duration": 0.007222, "end_time": "2026-06-13T02:50:58.303986+00:00", "exception": false, "start_time": "2026-06-13T02:50:58.296764+00:00", "status": "completed" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Date range: 2009-12-21 to 2017-11-16\n", "Unique assets: 39\n" ] }, { "data": { "text/html": [ "
| symbol | timestamp | news_date | trade_date | news_surprise | sentiment_mean | sentiment_std | weighted_surprise | sentiment_momentum | coverage_count | fwd_ret_1d | fwd_ret_5d | fwd_ret_20d |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| str | date | date | str | f64 | f64 | f64 | f64 | f64 | i64 | f64 | f64 | f64 |
| "ADBE" | 2010-04-30 | 2010-04-29 | "2010-04-30" | 0.222876 | 0.0 | 0.0 | 0.0 | 0.0 | 2 | -0.000298 | -0.035119 | -0.045238 |
| "ADBE" | 2010-05-04 | 2010-05-03 | "2010-05-04" | 0.516712 | 0.0 | 0.0 | 0.0 | -0.666667 | 1 | -0.005432 | 0.019916 | -0.006035 |
| "ADBE" | 2010-05-06 | 2010-05-05 | "2010-05-06" | 0.348653 | 0.0 | 0.0 | 0.0 | 0.0 | 1 | -0.002155 | 0.059095 | -0.027624 |
| "ADBE" | 2010-05-12 | 2010-05-11 | "2010-05-12" | 0.570604 | 0.0 | 0.0 | 0.0 | 0.0 | 1 | -0.011207 | -0.055172 | -0.088506 |
| "ADBE" | 2010-05-13 | 2010-05-12 | "2010-05-13" | 0.822824 | 0.0 | 0.0 | 0.0 | -1.0 | 1 | -0.020052 | -0.077594 | -0.077884 |
| "ADBE" | 2010-05-17 | 2010-05-14 | "2010-05-17" | 0.363815 | 0.0 | 0.0 | 0.0 | 0.0 | 1 | -0.015676 | -0.064478 | -0.03845 |
| "ADBE" | 2010-05-18 | 2010-05-17 | "2010-05-18" | 0.724458 | 0.0 | 0.0 | 0.0 | 0.0 | 1 | -0.012019 | -0.051983 | -0.02524 |
| "ADBE" | 2010-05-19 | 2010-05-18 | "2010-05-19" | 0.407025 | 0.0 | 0.0 | 0.0 | 0.0 | 1 | -0.034672 | -0.059002 | 0.007299 |
| "ADBE" | 2010-05-27 | 2010-05-26 | "2010-05-27" | 0.50654 | 0.0 | 0.0 | 0.0 | 0.0 | 2 | -0.003727 | -0.018866 | -0.072981 |
| "ADBE" | 2010-06-08 | 2010-06-07 | "2010-06-08" | 0.62012 | 0.0 | 0.0 | 0.0 | -1.0 | 1 | -0.00871 | 0.04871 | -0.135806 |