{ "cells": [ { "cell_type": "markdown", "id": "a7b01182", "metadata": { "papermill": { "duration": 0.004415, "end_time": "2026-06-14T17:03:27.628075+00:00", "exception": false, "start_time": "2026-06-14T17:03:27.623660+00:00", "status": "completed" } }, "source": [ "# Sentiment Analysis Evolution: TF-IDF → Word2Vec → Transformers\n", "\n", "**Chapter 10: From Text to Features - The Transformer Breakthrough**\n", "**Section Reference**: See Sections 10.2, 10.3, 10.5 for conceptual discussion\n", "\n", "**Docker image**: `ml4t-py312`\n", "\n", "> **Docker required**: This notebook uses `gensim`, which has no Python 3.14 support.\n", "> Run with:\n", "> ```bash\n", "> docker compose --profile py312 run --rm py312 python 10_text_feature_engineering/03_sentiment_evolution.py\n", "> ```\n", "\n", "## Purpose\n", "This notebook demonstrates the evolution of text representation for sentiment\n", "classification, comparing three approaches on the Financial PhraseBank dataset.\n", "We show how each generation of NLP techniques addresses limitations of its\n", "predecessors, culminating in Transformer-based models.\n", "\n", "## Learning Objectives\n", "After completing this notebook, you will be able to:\n", "- Implement TF-IDF vectorization for document classification\n", "- Use pre-trained static embeddings (GloVe) for document-level features\n", "- Apply a pre-trained Transformer (FinBERT) without task-specific fine-tuning\n", "- Compare accuracy and F1 scores across NLP paradigms\n", "- Interpret confusion matrices to identify class-specific errors\n", "- Understand why pre-trained models need fine-tuning for new tasks\n", "\n", "## Prerequisites\n", "- Sections 10.1–10.4 of the chapter (TF-IDF, static embeddings, Transformers).\n", "- Financial PhraseBank `sentences_allagree` subset on disk (loaded via\n", " `data.load_financial_phrasebank`).\n", "\n", "## Related Notebooks\n", "- `01_word2vec_training.py` — Skip-gram mechanics on the same corpus.\n", "- `04_bert_finetuning.py` — fine-tunes FinBERT on PhraseBank (this notebook\n", " shows the pre-fine-tuning baseline)." ] }, { "cell_type": "code", "execution_count": 1, "id": "1dc15c28", "metadata": { "execution": { "iopub.execute_input": "2026-06-14T17:03:27.637738Z", "iopub.status.busy": "2026-06-14T17:03:27.637595Z", "iopub.status.idle": "2026-06-14T17:03:37.595338Z", "shell.execute_reply": "2026-06-14T17:03:37.594742Z" }, "papermill": { "duration": 9.963966, "end_time": "2026-06-14T17:03:37.595982+00:00", "exception": false, "start_time": "2026-06-14T17:03:27.632016+00:00", "status": "completed" } }, "outputs": [], "source": [ "\"\"\"Sentiment Analysis Evolution — compare TF-IDF, Word2Vec, and Transformer approaches on Financial PhraseBank.\"\"\"\n", "\n", "import json\n", "import warnings\n", "\n", "import gensim.downloader as api\n", "import matplotlib.pyplot as plt\n", "import numpy as np\n", "import polars as pl\n", "import seaborn as sns\n", "import torch\n", "from sklearn.feature_extraction.text import TfidfVectorizer\n", "from sklearn.linear_model import LogisticRegression\n", "from sklearn.metrics import accuracy_score, confusion_matrix, f1_score\n", "from sklearn.model_selection import train_test_split\n", "from transformers import pipeline\n", "from transformers import set_seed as set_transformers_seed\n", "\n", "from data import load_financial_phrasebank as load_financial_phrasebank_canonical\n", "from utils.paths import get_chapter_dir\n", "from utils.reproducibility import set_global_seeds\n", "\n", "warnings.filterwarnings(\"ignore\", category=UserWarning)" ] }, { "cell_type": "code", "execution_count": 2, "id": "1bd2a9d8", "metadata": { "execution": { "iopub.execute_input": "2026-06-14T17:03:37.605088Z", "iopub.status.busy": "2026-06-14T17:03:37.604927Z", "iopub.status.idle": "2026-06-14T17:03:37.607174Z", "shell.execute_reply": "2026-06-14T17:03:37.606793Z" }, "papermill": { "duration": 0.007437, "end_time": "2026-06-14T17:03:37.607560+00:00", "exception": false, "start_time": "2026-06-14T17:03:37.600123+00:00", "status": "completed" }, "tags": [ "parameters" ] }, "outputs": [], "source": [ "# Production defaults — Papermill can override for fast CI runs.\n", "SEED = 42\n", "MAX_SAMPLES = 0 # 0 = use the full sentences_allagree subset\n", "FINBERT_TEST_SAMPLES = 0 # 0 = run FinBERT on the entire stratified test split" ] }, { "cell_type": "code", "execution_count": 3, "id": "b917d7f8", "metadata": { "execution": { "iopub.execute_input": "2026-06-14T17:03:37.616153Z", "iopub.status.busy": "2026-06-14T17:03:37.616028Z", "iopub.status.idle": "2026-06-14T17:03:37.863644Z", "shell.execute_reply": "2026-06-14T17:03:37.863062Z" }, "papermill": { "duration": 0.252618, "end_time": "2026-06-14T17:03:37.864108+00:00", "exception": false, "start_time": "2026-06-14T17:03:37.611490+00:00", "status": "completed" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "======================================================================\n", "EXPERIMENT CONFIGURATION\n", "======================================================================\n", "{\n", " \"random_seed\": 42,\n", " \"test_size\": 0.2,\n", " \"dataset\": {\n", " \"name\": \"takala/financial_phrasebank\",\n", " \"subset\": \"sentences_allagree\",\n", " \"description\": \"Financial PhraseBank \\u2014 100% annotator agreement subset (2,264 sentences)\"\n", " },\n", " \"tfidf\": {\n", " \"max_features\": 5000,\n", " \"ngram_range\": [\n", " 1,\n", " 2\n", " ],\n", " \"min_df\": 2\n", " },\n", " \"glove\": {\n", " \"model\": \"glove-wiki-gigaword-100\",\n", " \"dim\": 100\n", " },\n", " \"finbert\": {\n", " \"model_id\": \"yiyanghkust/finbert-tone\",\n", " \"description\": \"FinBERT fine-tuned on analyst reports for sentiment (NOT PhraseBank)\",\n", " \"tokenizer_id\": \"yiyanghkust/finbert-tone\",\n", " \"max_length\": 512,\n", " \"labels\": {\n", " \"Negative\": 0,\n", " \"Neutral\": 1,\n", " \"Positive\": 2\n", " }\n", " }\n", "}\n", "======================================================================\n" ] } ], "source": [ "# Reproducibility — set_global_seeds covers Python random / NumPy / Torch.\n", "# transformers has its own RNG (used by Trainer + pipelines) that needs explicit seeding.\n", "set_global_seeds(SEED)\n", "set_transformers_seed(SEED)\n", "\n", "CONFIG = {\n", " \"random_seed\": SEED,\n", " \"test_size\": 0.2,\n", " \"dataset\": {\n", " \"name\": \"takala/financial_phrasebank\",\n", " \"subset\": \"sentences_allagree\",\n", " \"description\": \"Financial PhraseBank — 100% annotator agreement subset (2,264 sentences)\",\n", " },\n", " \"tfidf\": {\n", " \"max_features\": 5000,\n", " \"ngram_range\": (1, 2),\n", " \"min_df\": 2,\n", " },\n", " \"glove\": {\n", " \"model\": \"glove-wiki-gigaword-100\",\n", " \"dim\": 100,\n", " },\n", " \"finbert\": {\n", " \"model_id\": \"yiyanghkust/finbert-tone\",\n", " \"description\": \"FinBERT fine-tuned on analyst reports for sentiment (NOT PhraseBank)\",\n", " \"tokenizer_id\": \"yiyanghkust/finbert-tone\",\n", " \"max_length\": 512,\n", " \"labels\": {\"Negative\": 0, \"Neutral\": 1, \"Positive\": 2},\n", " },\n", "}\n", "\n", "print(\"=\" * 70)\n", "print(\"EXPERIMENT CONFIGURATION\")\n", "print(\"=\" * 70)\n", "print(json.dumps(CONFIG, indent=2))\n", "print(\"=\" * 70)" ] }, { "cell_type": "markdown", "id": "f438543b", "metadata": { "papermill": { "duration": 0.003927, "end_time": "2026-06-14T17:03:37.872301+00:00", "exception": false, "start_time": "2026-06-14T17:03:37.868374+00:00", "status": "completed" } }, "source": [ "## 2. Load Financial PhraseBank Dataset\n", "\n", "The Financial PhraseBank (Malo et al., 2014) consists of English-language\n", "sentences from financial news, each labelled positive / negative / neutral\n", "by 5–8 annotators. We use the `sentences_allagree` subset — the 2,264\n", "sentences where every annotator picked the same label, i.e., the\n", "highest-precision portion of the corpus." ] }, { "cell_type": "code", "execution_count": 4, "id": "7eedd53d", "metadata": { "execution": { "iopub.execute_input": "2026-06-14T17:03:37.880964Z", "iopub.status.busy": "2026-06-14T17:03:37.880823Z", "iopub.status.idle": "2026-06-14T17:03:37.902296Z", "shell.execute_reply": "2026-06-14T17:03:37.901730Z" }, "papermill": { "duration": 0.026499, "end_time": "2026-06-14T17:03:37.902616+00:00", "exception": false, "start_time": "2026-06-14T17:03:37.876117+00:00", "status": "completed" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Loaded 2,264 sentences\n", "\n", "Label distribution:\n", "shape: (3, 2)\n", "┌───────┬──────┐\n", "│ label ┆ len │\n", "│ --- ┆ --- │\n", "│ i64 ┆ u32 │\n", "╞═══════╪══════╡\n", "│ 0 ┆ 303 │\n", "│ 1 ┆ 1391 │\n", "│ 2 ┆ 570 │\n", "└───────┴──────┘\n" ] } ], "source": [ "\n", "\n", "def load_financial_phrasebank() -> pl.DataFrame:\n", " \"\"\"Load Financial PhraseBank from canonical local storage (sentences_allagree).\"\"\"\n", " return load_financial_phrasebank_canonical()\n", "\n", "\n", "df = load_financial_phrasebank()\n", "print(f\"Loaded {len(df):,} sentences\")\n", "print(\"\\nLabel distribution:\")\n", "print(df.group_by(\"label\").len().sort(\"label\"))\n", "\n", "if MAX_SAMPLES > 0 and len(df) > MAX_SAMPLES:\n", " per_label = max(MAX_SAMPLES // df[\"label\"].n_unique(), 1)\n", " df = (\n", " df.sort([\"label\", \"sentence\"])\n", " .group_by(\"label\", maintain_order=True)\n", " .head(per_label)\n", " .sort(\"sentence\")\n", " )\n", " print(f\"Reduced dataset for test run: {len(df):,} sentences\")\n", "\n", "# Map numeric labels to text\n", "label_map = {0: \"negative\", 1: \"neutral\", 2: \"positive\"}\n", "df = df.with_columns(pl.col(\"label\").replace_strict(label_map).alias(\"sentiment\"))" ] }, { "cell_type": "code", "execution_count": 5, "id": "3d219ada", "metadata": { "execution": { "iopub.execute_input": "2026-06-14T17:03:37.911823Z", "iopub.status.busy": "2026-06-14T17:03:37.911690Z", "iopub.status.idle": "2026-06-14T17:03:37.917659Z", "shell.execute_reply": "2026-06-14T17:03:37.917234Z" }, "papermill": { "duration": 0.011178, "end_time": "2026-06-14T17:03:37.918088+00:00", "exception": false, "start_time": "2026-06-14T17:03:37.906910+00:00", "status": "completed" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "======================================================================\n", "DATASET SANITY CHECKS\n", "======================================================================\n", "\n", "1. CLASS DISTRIBUTION (Full Dataset)\n", "shape: (3, 2)\n", "┌───────┬──────┐\n", "│ label ┆ len │\n", "│ --- ┆ --- │\n", "│ i64 ┆ u32 │\n", "╞═══════╪══════╡\n", "│ 0 ┆ 303 │\n", "│ 1 ┆ 1391 │\n", "│ 2 ┆ 570 │\n", "└───────┴──────┘\n", "\n", "Majority class: neutral (label=1)\n", "Majority baseline accuracy: 61.4%\n", "\n", "2. LABEL MAPPING VERIFICATION\n", " Dataset labels → Our labels:\n", " 0 → negative\n", " 1 → neutral\n", " 2 → positive\n", " [OK] Label mapping verified\n", "\n", "3. SAMPLE SENTENCES BY CLASS\n", " negative: 'Jan. 6 -- Ford is struggling in the face of slowing truck and SUV sales and a su...'\n", " neutral: 'According to Gran , the company has no plans to move all production to Russia , ...'\n", " positive: 'For the last quarter of 2010 , Componenta 's net sales doubled to EUR131m from E...'\n", "\n", "======================================================================\n" ] } ], "source": [ "# ============================================================================\n", "# SANITY CHECKS\n", "# ============================================================================\n", "# These checks ensure the data and label mapping are correct before training.\n", "\n", "print(\"\\n\" + \"=\" * 70)\n", "print(\"DATASET SANITY CHECKS\")\n", "print(\"=\" * 70)\n", "\n", "# 1. Class distribution in full dataset\n", "print(\"\\n1. CLASS DISTRIBUTION (Full Dataset)\")\n", "class_counts = df.group_by(\"label\").len().sort(\"label\")\n", "print(class_counts)\n", "\n", "# Majority class baseline\n", "total = len(df)\n", "majority_row = class_counts.sort(\"len\", descending=True).row(0)\n", "majority_label = majority_row[0] # label column\n", "majority_count = majority_row[1] # len column\n", "majority_baseline = majority_count / total\n", "print(f\"\\nMajority class: {label_map[majority_label]} (label={majority_label})\")\n", "print(f\"Majority baseline accuracy: {majority_baseline:.1%}\")\n", "\n", "# 2. Label mapping verification\n", "print(\"\\n2. LABEL MAPPING VERIFICATION\")\n", "print(\" Dataset labels → Our labels:\")\n", "for numeric, text in label_map.items():\n", " print(f\" {numeric} → {text}\")\n", "\n", "# Assert label mapping matches FinBERT expectations\n", "finbert_label_map = {\"Negative\": 0, \"Neutral\": 1, \"Positive\": 2}\n", "assert label_map == {0: \"negative\", 1: \"neutral\", 2: \"positive\"}, \"Label mapping mismatch!\"\n", "print(\" [OK] Label mapping verified\")\n", "\n", "# 3. Sample sentences by class\n", "print(\"\\n3. SAMPLE SENTENCES BY CLASS\")\n", "for label_id, label_name in label_map.items():\n", " sample = df.filter(pl.col(\"label\") == label_id).head(1)[\"sentence\"][0]\n", " print(f\" {label_name}: '{sample[:80]}...'\")\n", "\n", "print(\"\\n\" + \"=\" * 70)" ] }, { "cell_type": "code", "execution_count": 6, "id": "ec4e50de", "metadata": { "execution": { "iopub.execute_input": "2026-06-14T17:03:37.927618Z", "iopub.status.busy": "2026-06-14T17:03:37.927488Z", "iopub.status.idle": "2026-06-14T17:03:37.932248Z", "shell.execute_reply": "2026-06-14T17:03:37.931825Z" }, "papermill": { "duration": 0.010127, "end_time": "2026-06-14T17:03:37.932501+00:00", "exception": false, "start_time": "2026-06-14T17:03:37.922374+00:00", "status": "completed" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "======================================================================\n", "TRAIN/TEST SPLIT DETAILS\n", "======================================================================\n", "Split protocol: Stratified random split (preserves class proportions)\n", "Test size: 0.2 (20%)\n", "Random seed: 42\n", "\n", "Train samples: 1,811\n", "Test samples: 453\n" ] } ], "source": [ "# Train/test split (convert Polars columns to numpy for sklearn)\n", "X_train, X_test, y_train, y_test = train_test_split(\n", " df[\"sentence\"].to_numpy(),\n", " df[\"label\"].to_numpy(),\n", " test_size=CONFIG[\"test_size\"],\n", " random_state=SEED,\n", " stratify=df[\"label\"].to_numpy(),\n", ")\n", "\n", "# Print split details\n", "print(\"\\n\" + \"=\" * 70)\n", "print(\"TRAIN/TEST SPLIT DETAILS\")\n", "print(\"=\" * 70)\n", "print(\"Split protocol: Stratified random split (preserves class proportions)\")\n", "print(f\"Test size: {CONFIG['test_size']} ({CONFIG['test_size'] * 100:.0f}%)\")\n", "print(f\"Random seed: {SEED}\")\n", "print(f\"\\nTrain samples: {len(X_train):,}\")\n", "print(f\"Test samples: {len(X_test):,}\")\n", "\n", "if FINBERT_TEST_SAMPLES > 0 and len(X_test) > FINBERT_TEST_SAMPLES:\n", " sample_idx = np.random.choice(len(X_test), FINBERT_TEST_SAMPLES, replace=False)\n", " X_test_finbert = X_test[sample_idx]\n", " y_test_finbert = y_test[sample_idx]\n", " print(f\"FinBERT evaluation sample: {len(X_test_finbert):,}\")\n", "else:\n", " X_test_finbert = X_test\n", " y_test_finbert = y_test" ] }, { "cell_type": "code", "execution_count": 7, "id": "7564c409", "metadata": { "execution": { "iopub.execute_input": "2026-06-14T17:03:37.941576Z", "iopub.status.busy": "2026-06-14T17:03:37.941452Z", "iopub.status.idle": "2026-06-14T17:03:37.946143Z", "shell.execute_reply": "2026-06-14T17:03:37.945751Z" }, "papermill": { "duration": 0.009879, "end_time": "2026-06-14T17:03:37.946610+00:00", "exception": false, "start_time": "2026-06-14T17:03:37.936731+00:00", "status": "completed" } }, "outputs": [ { "data": { "text/html": [ "
| class | train | test |
|---|---|---|
| str | i64 | i64 |
| "negative" | 242 | 61 |
| "neutral" | 1113 | 278 |
| "positive" | 456 | 114 |