wehub-resource-sync
2398 lines
82 KiB
Plaintext
2398 lines
82 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "6f9e0fa6",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.004362,
|
||
"end_time": "2026-06-13T03:10:23.592855+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T03:10:23.588493+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"# Entity Resolution: Matching Company Names to Identifiers\n",
|
||
"\n",
|
||
"**Chapter 4: Fundamental and Alternative Data**\n",
|
||
"**Docker image**: `ml4t`\n",
|
||
"**Section Reference**: See Section 4.2 for entity resolution concepts\n",
|
||
"\n",
|
||
"## Purpose\n",
|
||
"\n",
|
||
"Entity resolution is the keystone problem in multi-source data integration. Before any data\n",
|
||
"can be combined, we must correctly link disparate real-world names like \"IBM Corp\" and\n",
|
||
"\"International Business Machines\" to the same unique security identifier. This notebook\n",
|
||
"demonstrates hierarchical matching approaches from deterministic to ML-based.\n",
|
||
"\n",
|
||
"## Learning Objectives\n",
|
||
"\n",
|
||
"After completing this notebook, you will be able to:\n",
|
||
"- Understand the entity resolution problem and why it's critical\n",
|
||
"- Build a hierarchical matching approach: deterministic → probabilistic → ML-based\n",
|
||
"- Work with standard identifiers (LEI, CIK, FIGI, CUSIP, ISIN)\n",
|
||
"- Apply fuzzy string matching for inconsistent names\n",
|
||
"- Evaluate matching quality and handle edge cases\n",
|
||
"\n",
|
||
"## Cross-References\n",
|
||
"\n",
|
||
"- **Upstream**: Alternative data vendors, government contracts, web scraped data\n",
|
||
"- **Downstream**: Chapter 8 `fundamental_factors.py`, any multi-source analysis\n",
|
||
"- **Related**: `02_sec_filing_explorer.py` (CIK mapping)\n",
|
||
"\n",
|
||
"## Key Concepts\n",
|
||
"\n",
|
||
"- **Deterministic Matching**: Exact matches on strong identifiers (LEI, CIK, FIGI)\n",
|
||
"- **Probabilistic Matching**: Fuzzy string algorithms (Levenshtein, Jaro-Winkler)\n",
|
||
"- **ML-Based Matching**: Embeddings recover paraphrases and abbreviations; renames and subsidiaries still need a curated alias table\n",
|
||
"- **Master Security Database**: Central repository linking all identifiers"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 1,
|
||
"id": "5b20d754",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T03:10:23.602680Z",
|
||
"iopub.status.busy": "2026-06-13T03:10:23.602372Z",
|
||
"iopub.status.idle": "2026-06-13T03:10:23.985977Z",
|
||
"shell.execute_reply": "2026-06-13T03:10:23.985577Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.389651,
|
||
"end_time": "2026-06-13T03:10:23.986486+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T03:10:23.596835+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"\"\"\"Entity Resolution — match company names to identifiers using hierarchical deterministic and fuzzy matching.\"\"\"\n",
|
||
"\n",
|
||
"import os\n",
|
||
"import warnings\n",
|
||
"\n",
|
||
"warnings.filterwarnings(\"ignore\")\n",
|
||
"\n",
|
||
"\n",
|
||
"import numpy as np\n",
|
||
"import plotly.express as px\n",
|
||
"import polars as pl\n",
|
||
"from rapidfuzz import fuzz as rfuzz\n",
|
||
"from rapidfuzz import process as rprocess"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 2,
|
||
"id": "a297f019",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T03:10:23.990408Z",
|
||
"iopub.status.busy": "2026-06-13T03:10:23.990109Z",
|
||
"iopub.status.idle": "2026-06-13T03:10:23.992043Z",
|
||
"shell.execute_reply": "2026-06-13T03:10:23.991711Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.004464,
|
||
"end_time": "2026-06-13T03:10:23.992505+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T03:10:23.988041+00:00",
|
||
"status": "completed"
|
||
},
|
||
"tags": [
|
||
"parameters"
|
||
]
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Production defaults — Papermill injects overrides for CI"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "95c9a667",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.001535,
|
||
"end_time": "2026-06-13T03:10:23.995730+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T03:10:23.994195+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"## 1. The Entity Resolution Problem\n",
|
||
"\n",
|
||
"### Why This Matters\n",
|
||
"\n",
|
||
"Consider these real-world scenarios that break naive matching:\n",
|
||
"\n",
|
||
"| Source A | Source B | Same Company? |\n",
|
||
"|----------|----------|---------------|\n",
|
||
"| \"Apple Inc.\" | \"APPLE INC\" | Yes |\n",
|
||
"| \"Microsoft Corporation\" | \"MSFT\" | Yes |\n",
|
||
"| \"ZOOM Video Communications\" | \"Zoom Technologies Inc\" | **NO!** (ZOOM vs ZM) |\n",
|
||
"| \"Alphabet Inc.\" | \"Google LLC\" | Yes (subsidiary) |\n",
|
||
"| \"Meta Platforms Inc.\" | \"Facebook Inc.\" | Yes (renamed) |\n",
|
||
"\n",
|
||
"Getting this wrong can be catastrophic for your strategy."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 3,
|
||
"id": "a047f8fe",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T03:10:23.999967Z",
|
||
"iopub.status.busy": "2026-06-13T03:10:23.999794Z",
|
||
"iopub.status.idle": "2026-06-13T03:10:24.004216Z",
|
||
"shell.execute_reply": "2026-06-13T03:10:24.003912Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.007345,
|
||
"end_time": "2026-06-13T03:10:24.004659+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T03:10:23.997314+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/html": [
|
||
"<div><style>\n",
|
||
".dataframe > thead > tr,\n",
|
||
".dataframe > tbody > tr {\n",
|
||
" text-align: right;\n",
|
||
" white-space: pre-wrap;\n",
|
||
"}\n",
|
||
"</style>\n",
|
||
"<small>shape: (5, 3)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>source</th><th>company_name</th><th>ticker_if_available</th></tr><tr><td>str</td><td>str</td><td>str</td></tr></thead><tbody><tr><td>"SEC Filing"</td><td>"MICROSOFT CORPORATION"</td><td>null</td></tr><tr><td>"News Feed"</td><td>"Microsoft Corp."</td><td>null</td></tr><tr><td>"Alt Data"</td><td>"microsoft corp"</td><td>null</td></tr><tr><td>"Price Feed"</td><td>"MSFT"</td><td>"MSFT"</td></tr><tr><td>"Research"</td><td>"Microsoft (NASDAQ: MSFT)"</td><td>"MSFT"</td></tr></tbody></table></div>"
|
||
],
|
||
"text/plain": [
|
||
"shape: (5, 3)\n",
|
||
"┌────────────┬──────────────────────────┬─────────────────────┐\n",
|
||
"│ source ┆ company_name ┆ ticker_if_available │\n",
|
||
"│ --- ┆ --- ┆ --- │\n",
|
||
"│ str ┆ str ┆ str │\n",
|
||
"╞════════════╪══════════════════════════╪═════════════════════╡\n",
|
||
"│ SEC Filing ┆ MICROSOFT CORPORATION ┆ null │\n",
|
||
"│ News Feed ┆ Microsoft Corp. ┆ null │\n",
|
||
"│ Alt Data ┆ microsoft corp ┆ null │\n",
|
||
"│ Price Feed ┆ MSFT ┆ MSFT │\n",
|
||
"│ Research ┆ Microsoft (NASDAQ: MSFT) ┆ MSFT │\n",
|
||
"└────────────┴──────────────────────────┴─────────────────────┘"
|
||
]
|
||
},
|
||
"execution_count": 3,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"# Create sample messy data to demonstrate the problem\n",
|
||
"messy_company_names = pl.DataFrame(\n",
|
||
" {\n",
|
||
" \"source\": [\"SEC Filing\", \"News Feed\", \"Alt Data\", \"Price Feed\", \"Research\"],\n",
|
||
" \"company_name\": [\n",
|
||
" \"MICROSOFT CORPORATION\",\n",
|
||
" \"Microsoft Corp.\",\n",
|
||
" \"microsoft corp\",\n",
|
||
" \"MSFT\",\n",
|
||
" \"Microsoft (NASDAQ: MSFT)\",\n",
|
||
" ],\n",
|
||
" \"ticker_if_available\": [None, None, None, \"MSFT\", \"MSFT\"],\n",
|
||
" }\n",
|
||
")\n",
|
||
"\n",
|
||
"messy_company_names"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "d297be9d",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.001784,
|
||
"end_time": "2026-06-13T03:10:24.008213+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T03:10:24.006429+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"## 2. Standard Identifiers: The First Line of Defense\n",
|
||
"\n",
|
||
"When available, standard identifiers provide deterministic matching:\n",
|
||
"\n",
|
||
"| Identifier | Description | Coverage | Example |\n",
|
||
"|------------|-------------|----------|---------|\n",
|
||
"| **CIK** | SEC Central Index Key | US SEC filers | 0000789019 (MSFT) |\n",
|
||
"| **LEI** | Legal Entity Identifier | Global, 2.5M+ entities | INR2EJN1ERAN0W5ZP974 (MSFT) |\n",
|
||
"| **FIGI** | Financial Instrument Global Identifier | Global securities | BBG000BPH459 (MSFT) |\n",
|
||
"| **CUSIP** | US/Canada securities | US/Canada | 594918104 (MSFT) |\n",
|
||
"| **ISIN** | International Securities ID | Global | US5949181045 (MSFT) |\n",
|
||
"| **Ticker** | Exchange symbol | Exchange-specific | MSFT |"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 4,
|
||
"id": "a46870d9",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T03:10:24.011873Z",
|
||
"iopub.status.busy": "2026-06-13T03:10:24.011741Z",
|
||
"iopub.status.idle": "2026-06-13T03:10:24.014504Z",
|
||
"shell.execute_reply": "2026-06-13T03:10:24.014107Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.005655,
|
||
"end_time": "2026-06-13T03:10:24.015295+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T03:10:24.009640+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Create a reference database with standard identifiers\n",
|
||
"COMPANY_NAMES = [\n",
|
||
" \"Microsoft Corporation\",\n",
|
||
" \"Apple Inc.\",\n",
|
||
" \"Alphabet Inc.\",\n",
|
||
" \"Amazon.com Inc.\",\n",
|
||
" \"Meta Platforms Inc.\",\n",
|
||
" \"NVIDIA Corporation\",\n",
|
||
" \"Tesla Inc.\",\n",
|
||
" \"Berkshire Hathaway Inc.\",\n",
|
||
" \"JPMorgan Chase & Co.\",\n",
|
||
" \"Johnson & Johnson\",\n",
|
||
"]\n",
|
||
"TICKERS = [\"MSFT\", \"AAPL\", \"GOOGL\", \"AMZN\", \"META\", \"NVDA\", \"TSLA\", \"BRK.B\", \"JPM\", \"JNJ\"]\n",
|
||
"CIKS = [\n",
|
||
" \"0000789019\",\n",
|
||
" \"0000320193\",\n",
|
||
" \"0001652044\",\n",
|
||
" \"0001018724\",\n",
|
||
" \"0001326801\",\n",
|
||
" \"0001045810\",\n",
|
||
" \"0001318605\",\n",
|
||
" \"0001067983\",\n",
|
||
" \"0000019617\",\n",
|
||
" \"0000200406\",\n",
|
||
"]\n",
|
||
"EXCHANGES = [\"NASDAQ\"] * 7 + [\"NYSE\"] * 3"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 5,
|
||
"id": "18fd45d3",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T03:10:24.022171Z",
|
||
"iopub.status.busy": "2026-06-13T03:10:24.022062Z",
|
||
"iopub.status.idle": "2026-06-13T03:10:24.026000Z",
|
||
"shell.execute_reply": "2026-06-13T03:10:24.025338Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.008403,
|
||
"end_time": "2026-06-13T03:10:24.026443+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T03:10:24.018040+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/html": [
|
||
"<div><style>\n",
|
||
".dataframe > thead > tr,\n",
|
||
".dataframe > tbody > tr {\n",
|
||
" text-align: right;\n",
|
||
" white-space: pre-wrap;\n",
|
||
"}\n",
|
||
"</style>\n",
|
||
"<small>shape: (10, 4)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>company_name</th><th>ticker</th><th>cik</th><th>exchange</th></tr><tr><td>str</td><td>str</td><td>str</td><td>str</td></tr></thead><tbody><tr><td>"Microsoft Corporation"</td><td>"MSFT"</td><td>"0000789019"</td><td>"NASDAQ"</td></tr><tr><td>"Apple Inc."</td><td>"AAPL"</td><td>"0000320193"</td><td>"NASDAQ"</td></tr><tr><td>"Alphabet Inc."</td><td>"GOOGL"</td><td>"0001652044"</td><td>"NASDAQ"</td></tr><tr><td>"Amazon.com Inc."</td><td>"AMZN"</td><td>"0001018724"</td><td>"NASDAQ"</td></tr><tr><td>"Meta Platforms Inc."</td><td>"META"</td><td>"0001326801"</td><td>"NASDAQ"</td></tr><tr><td>"NVIDIA Corporation"</td><td>"NVDA"</td><td>"0001045810"</td><td>"NASDAQ"</td></tr><tr><td>"Tesla Inc."</td><td>"TSLA"</td><td>"0001318605"</td><td>"NASDAQ"</td></tr><tr><td>"Berkshire Hathaway Inc."</td><td>"BRK.B"</td><td>"0001067983"</td><td>"NYSE"</td></tr><tr><td>"JPMorgan Chase & Co."</td><td>"JPM"</td><td>"0000019617"</td><td>"NYSE"</td></tr><tr><td>"Johnson & Johnson"</td><td>"JNJ"</td><td>"0000200406"</td><td>"NYSE"</td></tr></tbody></table></div>"
|
||
],
|
||
"text/plain": [
|
||
"shape: (10, 4)\n",
|
||
"┌─────────────────────────┬────────┬────────────┬──────────┐\n",
|
||
"│ company_name ┆ ticker ┆ cik ┆ exchange │\n",
|
||
"│ --- ┆ --- ┆ --- ┆ --- │\n",
|
||
"│ str ┆ str ┆ str ┆ str │\n",
|
||
"╞═════════════════════════╪════════╪════════════╪══════════╡\n",
|
||
"│ Microsoft Corporation ┆ MSFT ┆ 0000789019 ┆ NASDAQ │\n",
|
||
"│ Apple Inc. ┆ AAPL ┆ 0000320193 ┆ NASDAQ │\n",
|
||
"│ Alphabet Inc. ┆ GOOGL ┆ 0001652044 ┆ NASDAQ │\n",
|
||
"│ Amazon.com Inc. ┆ AMZN ┆ 0001018724 ┆ NASDAQ │\n",
|
||
"│ Meta Platforms Inc. ┆ META ┆ 0001326801 ┆ NASDAQ │\n",
|
||
"│ NVIDIA Corporation ┆ NVDA ┆ 0001045810 ┆ NASDAQ │\n",
|
||
"│ Tesla Inc. ┆ TSLA ┆ 0001318605 ┆ NASDAQ │\n",
|
||
"│ Berkshire Hathaway Inc. ┆ BRK.B ┆ 0001067983 ┆ NYSE │\n",
|
||
"│ JPMorgan Chase & Co. ┆ JPM ┆ 0000019617 ┆ NYSE │\n",
|
||
"│ Johnson & Johnson ┆ JNJ ┆ 0000200406 ┆ NYSE │\n",
|
||
"└─────────────────────────┴────────┴────────────┴──────────┘"
|
||
]
|
||
},
|
||
"execution_count": 5,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"reference_securities = pl.DataFrame(\n",
|
||
" {\n",
|
||
" \"company_name\": COMPANY_NAMES,\n",
|
||
" \"ticker\": TICKERS,\n",
|
||
" \"cik\": CIKS,\n",
|
||
" \"exchange\": EXCHANGES,\n",
|
||
" }\n",
|
||
")\n",
|
||
"reference_securities"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "ac3210df",
|
||
"metadata": {
|
||
"lines_to_next_cell": 2,
|
||
"papermill": {
|
||
"duration": 0.001852,
|
||
"end_time": "2026-06-13T03:10:24.030367+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T03:10:24.028515+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"## 3. Stage 1: Deterministic Matching\n",
|
||
"\n",
|
||
"The first stage uses exact matches on strong identifiers."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 6,
|
||
"id": "972249dd",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T03:10:24.034813Z",
|
||
"iopub.status.busy": "2026-06-13T03:10:24.034579Z",
|
||
"iopub.status.idle": "2026-06-13T03:10:24.060106Z",
|
||
"shell.execute_reply": "2026-06-13T03:10:24.059384Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.028587,
|
||
"end_time": "2026-06-13T03:10:24.060685+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T03:10:24.032098+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/html": [
|
||
"<div><style>\n",
|
||
".dataframe > thead > tr,\n",
|
||
".dataframe > tbody > tr {\n",
|
||
" text-align: right;\n",
|
||
" white-space: pre-wrap;\n",
|
||
"}\n",
|
||
"</style>\n",
|
||
"<small>shape: (3, 5)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>filing_id</th><th>company_name</th><th>cik</th><th>matched_ticker</th><th>match_method</th></tr><tr><td>i64</td><td>str</td><td>str</td><td>str</td><td>str</td></tr></thead><tbody><tr><td>1</td><td>"MSFT INC"</td><td>"0000789019"</td><td>"MSFT"</td><td>"deterministic:cik"</td></tr><tr><td>2</td><td>"APPLE COMPUTER"</td><td>"0000320193"</td><td>"AAPL"</td><td>"deterministic:cik"</td></tr><tr><td>3</td><td>"UNKNOWN CORP"</td><td>"9999999999"</td><td>null</td><td>null</td></tr></tbody></table></div>"
|
||
],
|
||
"text/plain": [
|
||
"shape: (3, 5)\n",
|
||
"┌───────────┬────────────────┬────────────┬────────────────┬───────────────────┐\n",
|
||
"│ filing_id ┆ company_name ┆ cik ┆ matched_ticker ┆ match_method │\n",
|
||
"│ --- ┆ --- ┆ --- ┆ --- ┆ --- │\n",
|
||
"│ i64 ┆ str ┆ str ┆ str ┆ str │\n",
|
||
"╞═══════════╪════════════════╪════════════╪════════════════╪═══════════════════╡\n",
|
||
"│ 1 ┆ MSFT INC ┆ 0000789019 ┆ MSFT ┆ deterministic:cik │\n",
|
||
"│ 2 ┆ APPLE COMPUTER ┆ 0000320193 ┆ AAPL ┆ deterministic:cik │\n",
|
||
"│ 3 ┆ UNKNOWN CORP ┆ 9999999999 ┆ null ┆ null │\n",
|
||
"└───────────┴────────────────┴────────────┴────────────────┴───────────────────┘"
|
||
]
|
||
},
|
||
"execution_count": 6,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"def deterministic_match(\n",
|
||
" source_df: pl.DataFrame, reference_df: pl.DataFrame, match_columns: list[str]\n",
|
||
") -> pl.DataFrame:\n",
|
||
" \"\"\"Stage 1: Deterministic matching on exact identifier values.\"\"\"\n",
|
||
" result = source_df.clone()\n",
|
||
" result = result.with_columns(pl.lit(None).alias(\"matched_ticker\"))\n",
|
||
" result = result.with_columns(pl.lit(None).alias(\"match_method\"))\n",
|
||
"\n",
|
||
" for col in match_columns:\n",
|
||
" if col not in source_df.columns or col not in reference_df.columns:\n",
|
||
" continue\n",
|
||
"\n",
|
||
" # Find unmatched rows\n",
|
||
" unmatched_mask = result[\"matched_ticker\"].is_null()\n",
|
||
"\n",
|
||
" if unmatched_mask.sum() == 0:\n",
|
||
" break\n",
|
||
"\n",
|
||
" # Try to match on this column\n",
|
||
" # Dedupe reference to avoid row expansion on duplicate keys\n",
|
||
" ref_dedup = reference_df.select([col, \"ticker\"]).unique(subset=[col], keep=\"first\")\n",
|
||
" matches = source_df.join(ref_dedup, on=col, how=\"left\")\n",
|
||
"\n",
|
||
" # Update matched rows\n",
|
||
" result = result.with_columns(\n",
|
||
" [\n",
|
||
" pl.when(unmatched_mask & matches[\"ticker\"].is_not_null())\n",
|
||
" .then(matches[\"ticker\"])\n",
|
||
" .otherwise(result[\"matched_ticker\"])\n",
|
||
" .alias(\"matched_ticker\"),\n",
|
||
" pl.when(unmatched_mask & matches[\"ticker\"].is_not_null())\n",
|
||
" .then(pl.lit(f\"deterministic:{col}\"))\n",
|
||
" .otherwise(result[\"match_method\"])\n",
|
||
" .alias(\"match_method\"),\n",
|
||
" ]\n",
|
||
" )\n",
|
||
"\n",
|
||
" return result\n",
|
||
"\n",
|
||
"\n",
|
||
"# Example: matching on CIK\n",
|
||
"source_with_cik = pl.DataFrame(\n",
|
||
" {\n",
|
||
" \"filing_id\": [1, 2, 3],\n",
|
||
" \"company_name\": [\"MSFT INC\", \"APPLE COMPUTER\", \"UNKNOWN CORP\"],\n",
|
||
" \"cik\": [\"0000789019\", \"0000320193\", \"9999999999\"],\n",
|
||
" }\n",
|
||
")\n",
|
||
"\n",
|
||
"matched = deterministic_match(\n",
|
||
" source_with_cik, reference_securities, match_columns=[\"cik\", \"ticker\"]\n",
|
||
")\n",
|
||
"matched"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "f2d743da",
|
||
"metadata": {
|
||
"lines_to_next_cell": 2,
|
||
"papermill": {
|
||
"duration": 0.001737,
|
||
"end_time": "2026-06-13T03:10:24.065847+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T03:10:24.064110+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"## 4. Stage 2: Probabilistic Matching with Fuzzy Strings\n",
|
||
"\n",
|
||
"When identifiers aren't available, we use fuzzy string matching algorithms:\n",
|
||
"\n",
|
||
"- **Levenshtein Distance**: Edit distance (insertions, deletions, substitutions)\n",
|
||
"- **Jaro-Winkler**: Emphasizes prefix matches (good for company names)\n",
|
||
"- **Token Set Ratio**: Handles word reordering (\"Apple Inc\" vs \"Inc Apple\")\n",
|
||
"- **Partial Ratio**: Handles substrings (\"Microsoft\" in \"Microsoft Corporation\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 7,
|
||
"id": "b3f81af4",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T03:10:24.071408Z",
|
||
"iopub.status.busy": "2026-06-13T03:10:24.071217Z",
|
||
"iopub.status.idle": "2026-06-13T03:10:24.075366Z",
|
||
"shell.execute_reply": "2026-06-13T03:10:24.074587Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.008283,
|
||
"end_time": "2026-06-13T03:10:24.076407+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T03:10:24.068124+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"def normalize_company_name(name: str) -> str:\n",
|
||
" \"\"\"\n",
|
||
" Normalize company name for matching.\n",
|
||
"\n",
|
||
" Removes common suffixes, punctuation, and standardizes case.\n",
|
||
" \"\"\"\n",
|
||
" if not name:\n",
|
||
" return \"\"\n",
|
||
"\n",
|
||
" name = name.upper().strip()\n",
|
||
"\n",
|
||
" # Remove common suffixes\n",
|
||
" suffixes = [\n",
|
||
" \" INC.\",\n",
|
||
" \" INC\",\n",
|
||
" \" INCORPORATED\",\n",
|
||
" \" CORP.\",\n",
|
||
" \" CORP\",\n",
|
||
" \" CORPORATION\",\n",
|
||
" \" LLC\",\n",
|
||
" \" LLP\",\n",
|
||
" \" LP\",\n",
|
||
" \" LTD\",\n",
|
||
" \" LIMITED\",\n",
|
||
" \" CO.\",\n",
|
||
" \" CO\",\n",
|
||
" \" COMPANY\",\n",
|
||
" \" PLC\",\n",
|
||
" \" SA\",\n",
|
||
" \" AG\",\n",
|
||
" \" NV\",\n",
|
||
" \" SE\",\n",
|
||
" \" GROUP\",\n",
|
||
" \" HOLDINGS\",\n",
|
||
" ]\n",
|
||
" for suffix in suffixes:\n",
|
||
" if name.endswith(suffix):\n",
|
||
" name = name[: -len(suffix)]\n",
|
||
"\n",
|
||
" # Remove punctuation\n",
|
||
" name = name.replace(\",\", \"\").replace(\".\", \"\").replace(\"&\", \"AND\")\n",
|
||
"\n",
|
||
" # Remove extra whitespace\n",
|
||
" name = \" \".join(name.split())\n",
|
||
"\n",
|
||
" return name"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 8,
|
||
"id": "47edae9b",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T03:10:24.084264Z",
|
||
"iopub.status.busy": "2026-06-13T03:10:24.084165Z",
|
||
"iopub.status.idle": "2026-06-13T03:10:24.087054Z",
|
||
"shell.execute_reply": "2026-06-13T03:10:24.086390Z"
|
||
},
|
||
"lines_to_next_cell": 2,
|
||
"papermill": {
|
||
"duration": 0.007619,
|
||
"end_time": "2026-06-13T03:10:24.087931+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T03:10:24.080312+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"def prepare_candidates(candidates: list[str]) -> tuple[list[str], list[str]]:\n",
|
||
" \"\"\"\n",
|
||
" Pre-normalize candidate names for fuzzy matching.\n",
|
||
"\n",
|
||
" Call this ONCE before batch matching to avoid O(N×M) normalization cost.\n",
|
||
"\n",
|
||
" Returns\n",
|
||
" -------\n",
|
||
" tuple[list[str], list[str]]\n",
|
||
" (normalized_candidates, original_candidates)\n",
|
||
" \"\"\"\n",
|
||
" return [normalize_company_name(c) for c in candidates], candidates"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "bfac9fbd",
|
||
"metadata": {
|
||
"lines_to_next_cell": 2,
|
||
"papermill": {
|
||
"duration": 0.002406,
|
||
"end_time": "2026-06-13T03:10:24.092615+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T03:10:24.090209+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"### Fuzzy Match Company\n",
|
||
"Find the best fuzzy match for a company name against a list of candidates."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 9,
|
||
"id": "3526478d",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T03:10:24.099149Z",
|
||
"iopub.status.busy": "2026-06-13T03:10:24.099045Z",
|
||
"iopub.status.idle": "2026-06-13T03:10:24.104515Z",
|
||
"shell.execute_reply": "2026-06-13T03:10:24.104012Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.009354,
|
||
"end_time": "2026-06-13T03:10:24.105161+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T03:10:24.095807+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Fuzzy Matching Results:\n",
|
||
"[OK] 'MICROSOFT CORP' -> 'Microsoft Corporation' (score: 100)\n",
|
||
"[OK] 'Microsoft Corporation Inc' -> 'Microsoft Corporation' (score: 100)\n",
|
||
"[FAIL] 'msft' -> 'None' (score: 0)\n",
|
||
"[OK] 'Apple Computer' -> 'Apple Inc.' (score: 100)\n",
|
||
"[OK] 'ALPHABET INC CLASS A' -> 'Alphabet Inc.' (score: 100)\n",
|
||
"[FAIL] 'Google LLC' -> 'None' (score: 0)\n",
|
||
"[FAIL] 'Amazon Web Services' -> 'None' (score: 0)\n",
|
||
"[FAIL] 'Facebook Inc' -> 'None' (score: 0)\n",
|
||
"[OK] 'NVIDIA CORP' -> 'NVIDIA Corporation' (score: 100)\n",
|
||
"[OK] 'Tesla Motors' -> 'Tesla Inc.' (score: 100)\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"def fuzzy_match_company(\n",
|
||
" query: str,\n",
|
||
" candidates: list[str],\n",
|
||
" threshold: int = 80,\n",
|
||
" method: str = \"token_set_ratio\",\n",
|
||
" candidates_norm: list[str] | None = None,\n",
|
||
") -> tuple[str | None, int]:\n",
|
||
" \"\"\"Find best fuzzy match for a company name. Returns (match, score) or (None, 0).\"\"\"\n",
|
||
" query_norm = normalize_company_name(query)\n",
|
||
"\n",
|
||
" # Use pre-normalized candidates if provided, otherwise normalize (slower for batch)\n",
|
||
" if candidates_norm is None:\n",
|
||
" candidates_norm = [normalize_company_name(c) for c in candidates]\n",
|
||
"\n",
|
||
" # Build index lookup for O(1) access (first occurrence wins for duplicates)\n",
|
||
" norm_to_first_idx: dict[str, int] = {}\n",
|
||
" for i, n in enumerate(candidates_norm):\n",
|
||
" if n not in norm_to_first_idx:\n",
|
||
" norm_to_first_idx[n] = i\n",
|
||
"\n",
|
||
" scorer = {\n",
|
||
" \"ratio\": rfuzz.ratio,\n",
|
||
" \"partial_ratio\": rfuzz.partial_ratio,\n",
|
||
" \"token_sort_ratio\": rfuzz.token_sort_ratio,\n",
|
||
" \"token_set_ratio\": rfuzz.token_set_ratio,\n",
|
||
" }.get(method, rfuzz.token_set_ratio)\n",
|
||
"\n",
|
||
" result = rprocess.extractOne(query_norm, candidates_norm, scorer=scorer, score_cutoff=threshold)\n",
|
||
"\n",
|
||
" if result:\n",
|
||
" idx = norm_to_first_idx[result[0]]\n",
|
||
" return candidates[idx], int(result[1])\n",
|
||
"\n",
|
||
" return None, 0\n",
|
||
"\n",
|
||
"\n",
|
||
"# Test fuzzy matching\n",
|
||
"test_names = [\n",
|
||
" \"MICROSOFT CORP\",\n",
|
||
" \"Microsoft Corporation Inc\",\n",
|
||
" \"msft\",\n",
|
||
" \"Apple Computer\",\n",
|
||
" \"ALPHABET INC CLASS A\",\n",
|
||
" \"Google LLC\", # Subsidiary - harder to match\n",
|
||
" \"Amazon Web Services\", # Subsidiary\n",
|
||
" \"Facebook Inc\", # Old name\n",
|
||
" \"NVIDIA CORP\",\n",
|
||
" \"Tesla Motors\", # Old name\n",
|
||
"]\n",
|
||
"\n",
|
||
"reference_names = reference_securities[\"company_name\"].to_list()\n",
|
||
"\n",
|
||
"print(\"Fuzzy Matching Results:\")\n",
|
||
"for name in test_names:\n",
|
||
" match, score = fuzzy_match_company(name, reference_names, threshold=70)\n",
|
||
" status = \"[OK] \" if match else \"[FAIL]\"\n",
|
||
" print(f\"{status} {name!r:42} -> {str(match)!r:28} (score: {score})\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "fd30ad72",
|
||
"metadata": {
|
||
"lines_to_next_cell": 2,
|
||
"papermill": {
|
||
"duration": 0.002014,
|
||
"end_time": "2026-06-13T03:10:24.111045+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T03:10:24.109031+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"## 5. Stage 3: ML-Based Matching with Embeddings\n",
|
||
"\n",
|
||
"Fuzzy scoring compares *surface forms*: it rewards shared tokens and characters.\n",
|
||
"That leaves two gaps. A ticker used as a name (\"msft\") shares no characters with\n",
|
||
"\"Microsoft Corporation\", and a paraphrase like \"Amazon Web Services\" shares no\n",
|
||
"token with \"Amazon.com\" once suffixes are stripped — both fall through fuzzy\n",
|
||
"matching entirely. A sentence-embedding model maps each name to a vector whose\n",
|
||
"neighbors are *semantically* related, so cosine similarity can recover matches\n",
|
||
"that share meaning rather than spelling.\n",
|
||
"\n",
|
||
"We embed with `all-MiniLM-L6-v2`, a small model that runs locally in well under\n",
|
||
"a second on this data, and match each query to its nearest reference name. The\n",
|
||
"model loads from the local Hugging Face cache; `HF_HUB_OFFLINE=1` keeps it off\n",
|
||
"the network. Where the model is not cached — a minimal CI image, for instance —\n",
|
||
"the cell skips Stage 3 and the deterministic and fuzzy stages stand on their own."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 10,
|
||
"id": "7c4a3e19",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T03:10:24.115143Z",
|
||
"iopub.status.busy": "2026-06-13T03:10:24.115035Z",
|
||
"iopub.status.idle": "2026-06-13T03:10:27.216528Z",
|
||
"shell.execute_reply": "2026-06-13T03:10:27.215862Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 3.104485,
|
||
"end_time": "2026-06-13T03:10:27.217136+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T03:10:24.112651+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"os.environ.setdefault(\"HF_HUB_OFFLINE\", \"1\")\n",
|
||
"os.environ.setdefault(\"TRANSFORMERS_OFFLINE\", \"1\")\n",
|
||
"\n",
|
||
"EMBED_MODEL = \"sentence-transformers/all-MiniLM-L6-v2\"\n",
|
||
"\n",
|
||
"embedding_model = None\n",
|
||
"try:\n",
|
||
" from sentence_transformers import SentenceTransformer\n",
|
||
"\n",
|
||
" embedding_model = SentenceTransformer(EMBED_MODEL)\n",
|
||
"except Exception as exc: # offline + uncached, or optional dependency missing\n",
|
||
" print(\n",
|
||
" f\"[skip] embedding model '{EMBED_MODEL}' unavailable ({type(exc).__name__}); \"\n",
|
||
" \"showing deterministic + fuzzy stages only. Pre-cache the model to run Stage 3.\"\n",
|
||
" )\n",
|
||
"\n",
|
||
"\n",
|
||
"def embedding_match(\n",
|
||
" query: str,\n",
|
||
" candidates: list[str],\n",
|
||
" model,\n",
|
||
" candidate_embeddings: np.ndarray | None = None,\n",
|
||
" candidates_norm: list[str] | None = None,\n",
|
||
") -> tuple[str | None, float]:\n",
|
||
" \"\"\"Stage 3: match a name to its nearest reference by embedding cosine similarity.\n",
|
||
"\n",
|
||
" Names are normalized first so suffix noise (\"Inc.\", \"Corp.\") does not dominate\n",
|
||
" the vector. Embeddings are L2-normalized, so their dot product is the cosine.\n",
|
||
" \"\"\"\n",
|
||
" if model is None:\n",
|
||
" return None, 0.0\n",
|
||
"\n",
|
||
" if candidates_norm is None:\n",
|
||
" candidates_norm = [normalize_company_name(c) for c in candidates]\n",
|
||
" if candidate_embeddings is None:\n",
|
||
" candidate_embeddings = model.encode(candidates_norm, normalize_embeddings=True)\n",
|
||
"\n",
|
||
" query_emb = model.encode([normalize_company_name(query)], normalize_embeddings=True)[0]\n",
|
||
" sims = candidate_embeddings @ query_emb\n",
|
||
" best = int(np.argmax(sims))\n",
|
||
" return candidates[best], float(sims[best])"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 11,
|
||
"id": "ef02ac68",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T03:10:27.224112Z",
|
||
"iopub.status.busy": "2026-06-13T03:10:27.223844Z",
|
||
"iopub.status.idle": "2026-06-13T03:10:27.464543Z",
|
||
"shell.execute_reply": "2026-06-13T03:10:27.464009Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.244696,
|
||
"end_time": "2026-06-13T03:10:27.464985+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T03:10:27.220289+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Query Fuzzy Embedding cos\n",
|
||
"----------------------------------------------------------------------------\n",
|
||
"MICROSOFT CORP Microsoft Corporation Microsoft Corporation 1.00\n",
|
||
"Microsoft Corporation Inc Microsoft Corporation Microsoft Corporation 1.00\n",
|
||
"msft None Microsoft Corporation 0.53\n",
|
||
"Apple Computer Apple Inc. Apple Inc. 0.83\n",
|
||
"ALPHABET INC CLASS A Alphabet Inc. Alphabet Inc. 0.72\n",
|
||
"Google LLC None Microsoft Corporation 0.38\n",
|
||
"Amazon Web Services None Amazon.com Inc. 0.65\n",
|
||
"Facebook Inc None Meta Platforms Inc. 0.35\n",
|
||
"NVIDIA CORP NVIDIA Corporation NVIDIA Corporation 1.00\n",
|
||
"Tesla Motors Tesla Inc. Tesla Inc. 0.84\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Compare fuzzy and embedding matches on the same queries, side by side.\n",
|
||
"if embedding_model is not None:\n",
|
||
" ref_norm = [normalize_company_name(c) for c in reference_names]\n",
|
||
" ref_embeddings = embedding_model.encode(ref_norm, normalize_embeddings=True)\n",
|
||
"\n",
|
||
" print(f\"{'Query':<27}{'Fuzzy':<22}{'Embedding':<22}{'cos':>5}\")\n",
|
||
" print(\"-\" * 76)\n",
|
||
" for name in test_names:\n",
|
||
" fz_match, _ = fuzzy_match_company(name, reference_names, threshold=70)\n",
|
||
" em_match, em_score = embedding_match(\n",
|
||
" name,\n",
|
||
" reference_names,\n",
|
||
" embedding_model,\n",
|
||
" candidate_embeddings=ref_embeddings,\n",
|
||
" candidates_norm=ref_norm,\n",
|
||
" )\n",
|
||
" print(f\"{name:<27}{str(fz_match):<22}{str(em_match):<22}{em_score:>5.2f}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "9b74d7c3",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.001842,
|
||
"end_time": "2026-06-13T03:10:27.468410+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T03:10:27.466568+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"### What embeddings add — and where they mislead\n",
|
||
"\n",
|
||
"Two queries that fuzzy matching drops entirely come back with the embedding:\n",
|
||
"\n",
|
||
"- **`Amazon Web Services` → Amazon.com** (cos ≈ 0.65): no shared token survives\n",
|
||
" normalization, so fuzzy returns nothing, but the names are semantically close.\n",
|
||
"- **`msft` → Microsoft** (cos ≈ 0.53): a ticker carries none of the characters of\n",
|
||
" the company name; the embedding still places it nearest the right entity.\n",
|
||
"\n",
|
||
"The other recovered names — `Apple Computer`, `Tesla Motors`, `Alphabet Inc Class\n",
|
||
"A` — fuzzy already matches, because the canonical name is a token subset. The\n",
|
||
"embedding agrees there; it earns its place on the two cases above.\n",
|
||
"\n",
|
||
"The renames and subsidiaries are where the method breaks, and the scores show\n",
|
||
"exactly why:\n",
|
||
"\n",
|
||
"- **`Google LLC` → Microsoft** (cos ≈ 0.38) is simply **wrong**. \"Google\" and\n",
|
||
" \"Alphabet\" share no linguistic similarity; the parent–subsidiary link is a\n",
|
||
" corporate fact, not a property of the strings.\n",
|
||
"- **`Facebook Inc` → Meta Platforms** (cos ≈ 0.35) is *correct* — yet it scores\n",
|
||
" **below the wrong Google match**. The signal that should flag a rename is\n",
|
||
" weaker than the signal behind an outright error, so no single threshold\n",
|
||
" separates the good low-confidence match from the bad one.\n",
|
||
"\n",
|
||
"That non-separability is the lesson. A confident wrong match silently corrupts\n",
|
||
"every downstream join, and the model is *least* reliable on exactly the cases —\n",
|
||
"renames, subsidiaries, ticker reassignments — that a master database exists to\n",
|
||
"track. Embeddings extend the probabilistic stage for paraphrases and\n",
|
||
"abbreviations; they do not replace a curated alias layer keyed to the security\n",
|
||
"master, where each such mapping is recorded deliberately rather than inferred."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "71a4fa07",
|
||
"metadata": {
|
||
"lines_to_next_cell": 2,
|
||
"papermill": {
|
||
"duration": 0.001415,
|
||
"end_time": "2026-06-13T03:10:27.471331+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T03:10:27.469916+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"## 6. Building a Master Security Database\n",
|
||
"\n",
|
||
"A production-grade entity resolution system maintains a master database that:\n",
|
||
"1. Links all identifier types\n",
|
||
"2. Tracks name changes over time\n",
|
||
"3. Maps subsidiaries to parents\n",
|
||
"4. Stores confidence scores for probabilistic matches"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 12,
|
||
"id": "71660030",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T03:10:27.475325Z",
|
||
"iopub.status.busy": "2026-06-13T03:10:27.475182Z",
|
||
"iopub.status.idle": "2026-06-13T03:10:27.486516Z",
|
||
"shell.execute_reply": "2026-06-13T03:10:27.485895Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.01412,
|
||
"end_time": "2026-06-13T03:10:27.486982+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T03:10:27.472862+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Master Database Resolution Results:\n",
|
||
"[OK] '0000789019' -> Microsoft Corporation (deterministic:cik, conf: 100)\n",
|
||
"[OK] 'Facebook' -> Meta Platforms Inc. (fuzzy:name_variant, conf: 100)\n",
|
||
"[OK] 'Google LLC' -> Alphabet Inc. (fuzzy:name_variant, conf: 100)\n",
|
||
"[OK] 'MSFT' -> Microsoft Corporation (fuzzy:name_variant, conf: 100)\n",
|
||
"[FAIL] 'Unknown Corp' -> No match\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"class MasterSecurityDatabase:\n",
|
||
" \"\"\"\n",
|
||
" Master Security Database for entity resolution.\n",
|
||
"\n",
|
||
" Maintains canonical mappings between company names and identifiers,\n",
|
||
" with support for name changes, subsidiaries, and confidence tracking.\n",
|
||
"\n",
|
||
" Note: Uses list accumulation internally to avoid O(N²) memory copying\n",
|
||
" from iterative pl.concat(). DataFrames are materialized lazily.\n",
|
||
" \"\"\"\n",
|
||
"\n",
|
||
" def __init__(self):\n",
|
||
" # Use list accumulation to avoid O(N²) concat anti-pattern\n",
|
||
" self._entity_storage: list[dict] = []\n",
|
||
" self._variant_storage: list[dict] = []\n",
|
||
" self._entities_df: pl.DataFrame | None = None\n",
|
||
" self._variants_df: pl.DataFrame | None = None\n",
|
||
" self._next_id = 1\n",
|
||
"\n",
|
||
" @property\n",
|
||
" def entities(self) -> pl.DataFrame:\n",
|
||
" \"\"\"Lazily materialize entities DataFrame.\"\"\"\n",
|
||
" if self._entities_df is None or len(self._entity_storage) > 0:\n",
|
||
" if self._entity_storage:\n",
|
||
" new_df = pl.DataFrame(self._entity_storage)\n",
|
||
" if self._entities_df is not None:\n",
|
||
" self._entities_df = pl.concat([self._entities_df, new_df])\n",
|
||
" else:\n",
|
||
" self._entities_df = new_df\n",
|
||
" self._entity_storage = []\n",
|
||
" elif self._entities_df is None:\n",
|
||
" # Return empty DataFrame with correct schema\n",
|
||
" self._entities_df = pl.DataFrame(\n",
|
||
" schema={\n",
|
||
" \"entity_id\": pl.Int64,\n",
|
||
" \"canonical_name\": pl.Utf8,\n",
|
||
" \"ticker\": pl.Utf8,\n",
|
||
" \"cik\": pl.Utf8,\n",
|
||
" \"lei\": pl.Utf8,\n",
|
||
" \"figi\": pl.Utf8,\n",
|
||
" \"parent_entity_id\": pl.Int64,\n",
|
||
" \"is_active\": pl.Boolean,\n",
|
||
" }\n",
|
||
" )\n",
|
||
" return self._entities_df\n",
|
||
"\n",
|
||
" @property\n",
|
||
" def name_variants(self) -> pl.DataFrame:\n",
|
||
" \"\"\"Lazily materialize name_variants DataFrame.\"\"\"\n",
|
||
" if self._variants_df is None or len(self._variant_storage) > 0:\n",
|
||
" if self._variant_storage:\n",
|
||
" new_df = pl.DataFrame(self._variant_storage)\n",
|
||
" if self._variants_df is not None:\n",
|
||
" self._variants_df = pl.concat([self._variants_df, new_df])\n",
|
||
" else:\n",
|
||
" self._variants_df = new_df\n",
|
||
" self._variant_storage = []\n",
|
||
" elif self._variants_df is None:\n",
|
||
" # Return empty DataFrame with correct schema\n",
|
||
" self._variants_df = pl.DataFrame(\n",
|
||
" schema={\n",
|
||
" \"entity_id\": pl.Int64,\n",
|
||
" \"name_variant\": pl.Utf8,\n",
|
||
" \"valid_from\": pl.Date,\n",
|
||
" \"valid_to\": pl.Date,\n",
|
||
" \"is_primary\": pl.Boolean,\n",
|
||
" }\n",
|
||
" )\n",
|
||
" return self._variants_df\n",
|
||
"\n",
|
||
" def add_entity(\n",
|
||
" self,\n",
|
||
" canonical_name: str,\n",
|
||
" ticker: str,\n",
|
||
" cik: str = None,\n",
|
||
" lei: str = None,\n",
|
||
" figi: str = None,\n",
|
||
" name_variants: list[str] = None,\n",
|
||
" parent_entity_id: int = None,\n",
|
||
" ) -> int:\n",
|
||
" \"\"\"Add a new entity to the database.\"\"\"\n",
|
||
" entity_id = self._next_id\n",
|
||
" self._next_id += 1\n",
|
||
"\n",
|
||
" # Accumulate to list (O(1)) instead of concat (O(N))\n",
|
||
" self._entity_storage.append(\n",
|
||
" {\n",
|
||
" \"entity_id\": entity_id,\n",
|
||
" \"canonical_name\": canonical_name,\n",
|
||
" \"ticker\": ticker,\n",
|
||
" \"cik\": cik,\n",
|
||
" \"lei\": lei,\n",
|
||
" \"figi\": figi,\n",
|
||
" \"parent_entity_id\": parent_entity_id,\n",
|
||
" \"is_active\": True,\n",
|
||
" }\n",
|
||
" )\n",
|
||
"\n",
|
||
" # Add name variants\n",
|
||
" all_names = [canonical_name] + (name_variants or [])\n",
|
||
" for i, name in enumerate(all_names):\n",
|
||
" self._variant_storage.append(\n",
|
||
" {\n",
|
||
" \"entity_id\": entity_id,\n",
|
||
" \"name_variant\": name,\n",
|
||
" \"valid_from\": None,\n",
|
||
" \"valid_to\": None,\n",
|
||
" \"is_primary\": i == 0,\n",
|
||
" }\n",
|
||
" )\n",
|
||
"\n",
|
||
" return entity_id\n",
|
||
"\n",
|
||
" def resolve(self, query: str, identifier_type: str = None, threshold: int = 80) -> dict | None:\n",
|
||
" \"\"\"\n",
|
||
" Resolve a company name or identifier to canonical entity.\n",
|
||
"\n",
|
||
" Returns entity info or None if no match found.\n",
|
||
" \"\"\"\n",
|
||
" # Stage 1: Try deterministic match on identifier\n",
|
||
" if identifier_type and identifier_type in self.entities.columns:\n",
|
||
" matches = self.entities.filter(pl.col(identifier_type) == query)\n",
|
||
" if len(matches) > 0:\n",
|
||
" return {\n",
|
||
" \"entity_id\": matches[\"entity_id\"][0],\n",
|
||
" \"canonical_name\": matches[\"canonical_name\"][0],\n",
|
||
" \"ticker\": matches[\"ticker\"][0],\n",
|
||
" \"match_method\": f\"deterministic:{identifier_type}\",\n",
|
||
" \"confidence\": 100,\n",
|
||
" }\n",
|
||
"\n",
|
||
" # Stage 2: Try fuzzy match on name variants\n",
|
||
" all_variants = self.name_variants[\"name_variant\"].to_list()\n",
|
||
" match, score = fuzzy_match_company(query, all_variants, threshold=threshold)\n",
|
||
"\n",
|
||
" if match:\n",
|
||
" # Find entity for this variant\n",
|
||
" variant_row = self.name_variants.filter(pl.col(\"name_variant\") == match)\n",
|
||
" entity_id = variant_row[\"entity_id\"][0]\n",
|
||
" entity = self.entities.filter(pl.col(\"entity_id\") == entity_id)\n",
|
||
"\n",
|
||
" return {\n",
|
||
" \"entity_id\": entity_id,\n",
|
||
" \"canonical_name\": entity[\"canonical_name\"][0],\n",
|
||
" \"ticker\": entity[\"ticker\"][0],\n",
|
||
" \"match_method\": \"fuzzy:name_variant\",\n",
|
||
" \"confidence\": score,\n",
|
||
" \"matched_variant\": match,\n",
|
||
" }\n",
|
||
"\n",
|
||
" return None\n",
|
||
"\n",
|
||
"\n",
|
||
"# Build sample master database\n",
|
||
"msdb = MasterSecurityDatabase()\n",
|
||
"\n",
|
||
"# Add entities with name variants\n",
|
||
"msdb.add_entity(\n",
|
||
" \"Microsoft Corporation\",\n",
|
||
" \"MSFT\",\n",
|
||
" cik=\"0000789019\",\n",
|
||
" name_variants=[\"Microsoft Corp\", \"Microsoft Inc\", \"MSFT\"],\n",
|
||
")\n",
|
||
"\n",
|
||
"msdb.add_entity(\n",
|
||
" \"Apple Inc.\",\n",
|
||
" \"AAPL\",\n",
|
||
" cik=\"0000320193\",\n",
|
||
" name_variants=[\"Apple Computer\", \"Apple Computer Inc\", \"AAPL\"],\n",
|
||
")\n",
|
||
"\n",
|
||
"msdb.add_entity(\n",
|
||
" \"Meta Platforms Inc.\",\n",
|
||
" \"META\",\n",
|
||
" cik=\"0001326801\",\n",
|
||
" name_variants=[\"Facebook Inc\", \"Facebook\", \"Meta\", \"META\"],\n",
|
||
")\n",
|
||
"\n",
|
||
"msdb.add_entity(\n",
|
||
" \"Alphabet Inc.\",\n",
|
||
" \"GOOGL\",\n",
|
||
" cik=\"0001652044\",\n",
|
||
" name_variants=[\"Google Inc\", \"Google LLC\", \"GOOGL\", \"GOOG\"],\n",
|
||
")\n",
|
||
"\n",
|
||
"# Test resolution\n",
|
||
"test_queries = [\n",
|
||
" (\"0000789019\", \"cik\"), # Deterministic\n",
|
||
" (\"Facebook\", None), # Fuzzy - old name\n",
|
||
" (\"Google LLC\", None), # Fuzzy - subsidiary name\n",
|
||
" (\"MSFT\", None), # Fuzzy - ticker as name\n",
|
||
" (\"Unknown Corp\", None), # No match\n",
|
||
"]\n",
|
||
"\n",
|
||
"print(\"Master Database Resolution Results:\")\n",
|
||
"for query, id_type in test_queries:\n",
|
||
" result = msdb.resolve(query, identifier_type=id_type)\n",
|
||
" if result:\n",
|
||
" print(\n",
|
||
" f\"[OK] {query!r:18} -> {result['canonical_name']:24} \"\n",
|
||
" f\"({result['match_method']}, conf: {result['confidence']})\"\n",
|
||
" )\n",
|
||
" else:\n",
|
||
" print(f\"[FAIL] {query!r:18} -> No match\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "ecb18bd6",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.001624,
|
||
"end_time": "2026-06-13T03:10:27.490429+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T03:10:27.488805+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"## 7. Practical Application: Matching Alternative Data\n",
|
||
"\n",
|
||
"Real-world entity resolution must handle name changes (Facebook → Meta),\n",
|
||
"subsidiaries (Google LLC → Alphabet), ticker confusion (ZOOM vs ZM),\n",
|
||
"and special characters (AT&T, S&P Global).\n",
|
||
"\n",
|
||
"Let's apply entity resolution to match messy alternative data (e.g., job postings)\n",
|
||
"to our reference securities."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 13,
|
||
"id": "f1f3c741",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T03:10:27.494865Z",
|
||
"iopub.status.busy": "2026-06-13T03:10:27.494662Z",
|
||
"iopub.status.idle": "2026-06-13T03:10:27.502898Z",
|
||
"shell.execute_reply": "2026-06-13T03:10:27.502195Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.011541,
|
||
"end_time": "2026-06-13T03:10:27.503421+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T03:10:27.491880+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/html": [
|
||
"<div><style>\n",
|
||
".dataframe > thead > tr,\n",
|
||
".dataframe > tbody > tr {\n",
|
||
" text-align: right;\n",
|
||
" white-space: pre-wrap;\n",
|
||
"}\n",
|
||
"</style>\n",
|
||
"<small>shape: (10, 5)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>source_id</th><th>raw_company_name</th><th>matched_name</th><th>match_score</th><th>job_postings</th></tr><tr><td>i64</td><td>str</td><td>str</td><td>i64</td><td>i64</td></tr></thead><tbody><tr><td>1</td><td>"Microsoft Corporation - Redmon…</td><td>"Microsoft Corporation"</td><td>100</td><td>150</td></tr><tr><td>2</td><td>"APPLE INC"</td><td>"Apple Inc."</td><td>100</td><td>200</td></tr><tr><td>3</td><td>"GOOGLE"</td><td>null</td><td>0</td><td>180</td></tr><tr><td>4</td><td>"amazon.com"</td><td>"Amazon.com Inc."</td><td>100</td><td>300</td></tr><tr><td>5</td><td>"Facebook, Inc."</td><td>null</td><td>0</td><td>120</td></tr><tr><td>6</td><td>"NVIDIA Corp"</td><td>"NVIDIA Corporation"</td><td>100</td><td>80</td></tr><tr><td>7</td><td>"Tesla Motors Inc"</td><td>"Tesla Inc."</td><td>100</td><td>90</td></tr><tr><td>8</td><td>"Berkshire Hathaway"</td><td>"Berkshire Hathaway Inc."</td><td>100</td><td>50</td></tr><tr><td>9</td><td>"JP Morgan Chase"</td><td>"JPMorgan Chase & Co."</td><td>84</td><td>160</td></tr><tr><td>10</td><td>"J&J"</td><td>null</td><td>0</td><td>70</td></tr></tbody></table></div>"
|
||
],
|
||
"text/plain": [
|
||
"shape: (10, 5)\n",
|
||
"┌───────────┬─────────────────────────┬─────────────────────────┬─────────────┬──────────────┐\n",
|
||
"│ source_id ┆ raw_company_name ┆ matched_name ┆ match_score ┆ job_postings │\n",
|
||
"│ --- ┆ --- ┆ --- ┆ --- ┆ --- │\n",
|
||
"│ i64 ┆ str ┆ str ┆ i64 ┆ i64 │\n",
|
||
"╞═══════════╪═════════════════════════╪═════════════════════════╪═════════════╪══════════════╡\n",
|
||
"│ 1 ┆ Microsoft Corporation - ┆ Microsoft Corporation ┆ 100 ┆ 150 │\n",
|
||
"│ ┆ Redmon… ┆ ┆ ┆ │\n",
|
||
"│ 2 ┆ APPLE INC ┆ Apple Inc. ┆ 100 ┆ 200 │\n",
|
||
"│ 3 ┆ GOOGLE ┆ null ┆ 0 ┆ 180 │\n",
|
||
"│ 4 ┆ amazon.com ┆ Amazon.com Inc. ┆ 100 ┆ 300 │\n",
|
||
"│ 5 ┆ Facebook, Inc. ┆ null ┆ 0 ┆ 120 │\n",
|
||
"│ 6 ┆ NVIDIA Corp ┆ NVIDIA Corporation ┆ 100 ┆ 80 │\n",
|
||
"│ 7 ┆ Tesla Motors Inc ┆ Tesla Inc. ┆ 100 ┆ 90 │\n",
|
||
"│ 8 ┆ Berkshire Hathaway ┆ Berkshire Hathaway Inc. ┆ 100 ┆ 50 │\n",
|
||
"│ 9 ┆ JP Morgan Chase ┆ JPMorgan Chase & Co. ┆ 84 ┆ 160 │\n",
|
||
"│ 10 ┆ J&J ┆ null ┆ 0 ┆ 70 │\n",
|
||
"└───────────┴─────────────────────────┴─────────────────────────┴─────────────┴──────────────┘"
|
||
]
|
||
},
|
||
"execution_count": 13,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"# Simulate messy alternative data\n",
|
||
"alt_data_companies = pl.DataFrame(\n",
|
||
" {\n",
|
||
" \"source_id\": range(1, 11),\n",
|
||
" \"raw_company_name\": [\n",
|
||
" \"Microsoft Corporation - Redmond\",\n",
|
||
" \"APPLE INC\",\n",
|
||
" \"GOOGLE\",\n",
|
||
" \"amazon.com\",\n",
|
||
" \"Facebook, Inc.\",\n",
|
||
" \"NVIDIA Corp\",\n",
|
||
" \"Tesla Motors Inc\",\n",
|
||
" \"Berkshire Hathaway\",\n",
|
||
" \"JP Morgan Chase\",\n",
|
||
" \"J&J\",\n",
|
||
" ],\n",
|
||
" \"job_postings\": [150, 200, 180, 300, 120, 80, 90, 50, 160, 70],\n",
|
||
" }\n",
|
||
")\n",
|
||
"\n",
|
||
"\n",
|
||
"def batch_resolve(\n",
|
||
" df: pl.DataFrame, name_column: str, reference_names: list[str], threshold: int = 75\n",
|
||
") -> pl.DataFrame:\n",
|
||
" \"\"\"\n",
|
||
" Batch resolve company names in a dataframe.\n",
|
||
"\n",
|
||
" Pre-normalizes candidates once to avoid O(N×M) string operations.\n",
|
||
"\n",
|
||
" NOTE: This uses iter_rows() for educational clarity - it shows exactly what\n",
|
||
" happens per-row during entity resolution. For production systems with >10K rows,\n",
|
||
" consider using map_elements() or batch processing with vectorized operations.\n",
|
||
" The key optimization here is pre-normalizing candidates once outside the loop,\n",
|
||
" reducing complexity from O(N×M) to O(N+M).\n",
|
||
" \"\"\"\n",
|
||
" results = []\n",
|
||
"\n",
|
||
" # Pre-normalize candidates ONCE (not per-row) to avoid O(N×M) cost\n",
|
||
" candidates_norm, candidates_orig = prepare_candidates(reference_names)\n",
|
||
"\n",
|
||
" for row in df.iter_rows(named=True):\n",
|
||
" name = row[name_column]\n",
|
||
" match, score = fuzzy_match_company(\n",
|
||
" name, candidates_orig, threshold=threshold, candidates_norm=candidates_norm\n",
|
||
" )\n",
|
||
"\n",
|
||
" results.append(\n",
|
||
" {\n",
|
||
" **row,\n",
|
||
" \"matched_name\": match,\n",
|
||
" \"match_score\": score,\n",
|
||
" \"match_status\": \"matched\" if match else \"unmatched\",\n",
|
||
" }\n",
|
||
" )\n",
|
||
"\n",
|
||
" return pl.DataFrame(results)\n",
|
||
"\n",
|
||
"\n",
|
||
"# Resolve alternative data\n",
|
||
"alt_data_to_resolve = alt_data_companies\n",
|
||
"resolved_alt_data = batch_resolve(\n",
|
||
" alt_data_to_resolve,\n",
|
||
" \"raw_company_name\",\n",
|
||
" reference_securities[\"company_name\"].to_list(),\n",
|
||
" threshold=70,\n",
|
||
")\n",
|
||
"\n",
|
||
"resolved_alt_data.select(\n",
|
||
" [\"source_id\", \"raw_company_name\", \"matched_name\", \"match_score\", \"job_postings\"]\n",
|
||
")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "0899fb23",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.001672,
|
||
"end_time": "2026-06-13T03:10:27.507288+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T03:10:27.505616+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"## 8. Measuring Match Quality\n",
|
||
"\n",
|
||
"Always evaluate your entity resolution quality:\n",
|
||
"- **Precision**: What fraction of matches are correct?\n",
|
||
"- **Recall**: What fraction of true matches did we find?\n",
|
||
"- **F1 Score**: Harmonic mean of precision and recall"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 14,
|
||
"id": "0f69e62b",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T03:10:27.511270Z",
|
||
"iopub.status.busy": "2026-06-13T03:10:27.511128Z",
|
||
"iopub.status.idle": "2026-06-13T03:10:27.515278Z",
|
||
"shell.execute_reply": "2026-06-13T03:10:27.514732Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 0.007061,
|
||
"end_time": "2026-06-13T03:10:27.515924+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T03:10:27.508863+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/html": [
|
||
"<div><style>\n",
|
||
".dataframe > thead > tr,\n",
|
||
".dataframe > tbody > tr {\n",
|
||
" text-align: right;\n",
|
||
" white-space: pre-wrap;\n",
|
||
"}\n",
|
||
"</style>\n",
|
||
"<small>shape: (2, 3)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>match_status</th><th>count</th><th>avg_score</th></tr><tr><td>str</td><td>u32</td><td>f64</td></tr></thead><tbody><tr><td>"unmatched"</td><td>3</td><td>0.0</td></tr><tr><td>"matched"</td><td>7</td><td>97.714286</td></tr></tbody></table></div>"
|
||
],
|
||
"text/plain": [
|
||
"shape: (2, 3)\n",
|
||
"┌──────────────┬───────┬───────────┐\n",
|
||
"│ match_status ┆ count ┆ avg_score │\n",
|
||
"│ --- ┆ --- ┆ --- │\n",
|
||
"│ str ┆ u32 ┆ f64 │\n",
|
||
"╞══════════════╪═══════╪═══════════╡\n",
|
||
"│ unmatched ┆ 3 ┆ 0.0 │\n",
|
||
"│ matched ┆ 7 ┆ 97.714286 │\n",
|
||
"└──────────────┴───────┴───────────┘"
|
||
]
|
||
},
|
||
"execution_count": 14,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"match_stats = resolved_alt_data.group_by(\"match_status\").agg(\n",
|
||
" [pl.len().alias(\"count\"), pl.mean(\"match_score\").alias(\"avg_score\")]\n",
|
||
")\n",
|
||
"match_stats"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 15,
|
||
"id": "fc1183f4",
|
||
"metadata": {
|
||
"execution": {
|
||
"iopub.execute_input": "2026-06-13T03:10:27.521701Z",
|
||
"iopub.status.busy": "2026-06-13T03:10:27.521519Z",
|
||
"iopub.status.idle": "2026-06-13T03:10:28.683297Z",
|
||
"shell.execute_reply": "2026-06-13T03:10:28.682752Z"
|
||
},
|
||
"papermill": {
|
||
"duration": 1.165381,
|
||
"end_time": "2026-06-13T03:10:28.684143+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T03:10:27.518762+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"application/vnd.plotly.v1+json": {
|
||
"config": {
|
||
"plotlyServerURL": "https://plot.ly"
|
||
},
|
||
"data": [
|
||
{
|
||
"bingroup": "x",
|
||
"hovertemplate": "Fuzzy Match Score=%{x}<br>count=%{y}<extra></extra>",
|
||
"legendgroup": "",
|
||
"marker": {
|
||
"color": "#636efa",
|
||
"pattern": {
|
||
"shape": ""
|
||
}
|
||
},
|
||
"name": "",
|
||
"nbinsx": 20,
|
||
"orientation": "v",
|
||
"showlegend": false,
|
||
"type": "histogram",
|
||
"x": {
|
||
"bdata": "ZGQAZABkZGRUAA==",
|
||
"dtype": "i1"
|
||
},
|
||
"xaxis": "x",
|
||
"yaxis": "y"
|
||
}
|
||
],
|
||
"layout": {
|
||
"annotations": [
|
||
{
|
||
"showarrow": false,
|
||
"text": "Threshold (75)",
|
||
"x": 75,
|
||
"xanchor": "left",
|
||
"xref": "x",
|
||
"y": 1,
|
||
"yanchor": "top",
|
||
"yref": "y domain"
|
||
}
|
||
],
|
||
"barmode": "relative",
|
||
"legend": {
|
||
"tracegroupgap": 0
|
||
},
|
||
"shapes": [
|
||
{
|
||
"line": {
|
||
"color": "red",
|
||
"dash": "dash"
|
||
},
|
||
"type": "line",
|
||
"x0": 75,
|
||
"x1": 75,
|
||
"xref": "x",
|
||
"y0": 0,
|
||
"y1": 1,
|
||
"yref": "y domain"
|
||
}
|
||
],
|
||
"template": {
|
||
"data": {
|
||
"bar": [
|
||
{
|
||
"error_x": {
|
||
"color": "#2a3f5f"
|
||
},
|
||
"error_y": {
|
||
"color": "#2a3f5f"
|
||
},
|
||
"marker": {
|
||
"line": {
|
||
"color": "#E5ECF6",
|
||
"width": 0.5
|
||
},
|
||
"pattern": {
|
||
"fillmode": "overlay",
|
||
"size": 10,
|
||
"solidity": 0.2
|
||
}
|
||
},
|
||
"type": "bar"
|
||
}
|
||
],
|
||
"barpolar": [
|
||
{
|
||
"marker": {
|
||
"line": {
|
||
"color": "#E5ECF6",
|
||
"width": 0.5
|
||
},
|
||
"pattern": {
|
||
"fillmode": "overlay",
|
||
"size": 10,
|
||
"solidity": 0.2
|
||
}
|
||
},
|
||
"type": "barpolar"
|
||
}
|
||
],
|
||
"carpet": [
|
||
{
|
||
"aaxis": {
|
||
"endlinecolor": "#2a3f5f",
|
||
"gridcolor": "white",
|
||
"linecolor": "white",
|
||
"minorgridcolor": "white",
|
||
"startlinecolor": "#2a3f5f"
|
||
},
|
||
"baxis": {
|
||
"endlinecolor": "#2a3f5f",
|
||
"gridcolor": "white",
|
||
"linecolor": "white",
|
||
"minorgridcolor": "white",
|
||
"startlinecolor": "#2a3f5f"
|
||
},
|
||
"type": "carpet"
|
||
}
|
||
],
|
||
"choropleth": [
|
||
{
|
||
"colorbar": {
|
||
"outlinewidth": 0,
|
||
"ticks": ""
|
||
},
|
||
"type": "choropleth"
|
||
}
|
||
],
|
||
"contour": [
|
||
{
|
||
"colorbar": {
|
||
"outlinewidth": 0,
|
||
"ticks": ""
|
||
},
|
||
"colorscale": [
|
||
[
|
||
0.0,
|
||
"#0d0887"
|
||
],
|
||
[
|
||
0.1111111111111111,
|
||
"#46039f"
|
||
],
|
||
[
|
||
0.2222222222222222,
|
||
"#7201a8"
|
||
],
|
||
[
|
||
0.3333333333333333,
|
||
"#9c179e"
|
||
],
|
||
[
|
||
0.4444444444444444,
|
||
"#bd3786"
|
||
],
|
||
[
|
||
0.5555555555555556,
|
||
"#d8576b"
|
||
],
|
||
[
|
||
0.6666666666666666,
|
||
"#ed7953"
|
||
],
|
||
[
|
||
0.7777777777777778,
|
||
"#fb9f3a"
|
||
],
|
||
[
|
||
0.8888888888888888,
|
||
"#fdca26"
|
||
],
|
||
[
|
||
1.0,
|
||
"#f0f921"
|
||
]
|
||
],
|
||
"type": "contour"
|
||
}
|
||
],
|
||
"contourcarpet": [
|
||
{
|
||
"colorbar": {
|
||
"outlinewidth": 0,
|
||
"ticks": ""
|
||
},
|
||
"type": "contourcarpet"
|
||
}
|
||
],
|
||
"heatmap": [
|
||
{
|
||
"colorbar": {
|
||
"outlinewidth": 0,
|
||
"ticks": ""
|
||
},
|
||
"colorscale": [
|
||
[
|
||
0.0,
|
||
"#0d0887"
|
||
],
|
||
[
|
||
0.1111111111111111,
|
||
"#46039f"
|
||
],
|
||
[
|
||
0.2222222222222222,
|
||
"#7201a8"
|
||
],
|
||
[
|
||
0.3333333333333333,
|
||
"#9c179e"
|
||
],
|
||
[
|
||
0.4444444444444444,
|
||
"#bd3786"
|
||
],
|
||
[
|
||
0.5555555555555556,
|
||
"#d8576b"
|
||
],
|
||
[
|
||
0.6666666666666666,
|
||
"#ed7953"
|
||
],
|
||
[
|
||
0.7777777777777778,
|
||
"#fb9f3a"
|
||
],
|
||
[
|
||
0.8888888888888888,
|
||
"#fdca26"
|
||
],
|
||
[
|
||
1.0,
|
||
"#f0f921"
|
||
]
|
||
],
|
||
"type": "heatmap"
|
||
}
|
||
],
|
||
"histogram": [
|
||
{
|
||
"marker": {
|
||
"pattern": {
|
||
"fillmode": "overlay",
|
||
"size": 10,
|
||
"solidity": 0.2
|
||
}
|
||
},
|
||
"type": "histogram"
|
||
}
|
||
],
|
||
"histogram2d": [
|
||
{
|
||
"colorbar": {
|
||
"outlinewidth": 0,
|
||
"ticks": ""
|
||
},
|
||
"colorscale": [
|
||
[
|
||
0.0,
|
||
"#0d0887"
|
||
],
|
||
[
|
||
0.1111111111111111,
|
||
"#46039f"
|
||
],
|
||
[
|
||
0.2222222222222222,
|
||
"#7201a8"
|
||
],
|
||
[
|
||
0.3333333333333333,
|
||
"#9c179e"
|
||
],
|
||
[
|
||
0.4444444444444444,
|
||
"#bd3786"
|
||
],
|
||
[
|
||
0.5555555555555556,
|
||
"#d8576b"
|
||
],
|
||
[
|
||
0.6666666666666666,
|
||
"#ed7953"
|
||
],
|
||
[
|
||
0.7777777777777778,
|
||
"#fb9f3a"
|
||
],
|
||
[
|
||
0.8888888888888888,
|
||
"#fdca26"
|
||
],
|
||
[
|
||
1.0,
|
||
"#f0f921"
|
||
]
|
||
],
|
||
"type": "histogram2d"
|
||
}
|
||
],
|
||
"histogram2dcontour": [
|
||
{
|
||
"colorbar": {
|
||
"outlinewidth": 0,
|
||
"ticks": ""
|
||
},
|
||
"colorscale": [
|
||
[
|
||
0.0,
|
||
"#0d0887"
|
||
],
|
||
[
|
||
0.1111111111111111,
|
||
"#46039f"
|
||
],
|
||
[
|
||
0.2222222222222222,
|
||
"#7201a8"
|
||
],
|
||
[
|
||
0.3333333333333333,
|
||
"#9c179e"
|
||
],
|
||
[
|
||
0.4444444444444444,
|
||
"#bd3786"
|
||
],
|
||
[
|
||
0.5555555555555556,
|
||
"#d8576b"
|
||
],
|
||
[
|
||
0.6666666666666666,
|
||
"#ed7953"
|
||
],
|
||
[
|
||
0.7777777777777778,
|
||
"#fb9f3a"
|
||
],
|
||
[
|
||
0.8888888888888888,
|
||
"#fdca26"
|
||
],
|
||
[
|
||
1.0,
|
||
"#f0f921"
|
||
]
|
||
],
|
||
"type": "histogram2dcontour"
|
||
}
|
||
],
|
||
"mesh3d": [
|
||
{
|
||
"colorbar": {
|
||
"outlinewidth": 0,
|
||
"ticks": ""
|
||
},
|
||
"type": "mesh3d"
|
||
}
|
||
],
|
||
"parcoords": [
|
||
{
|
||
"line": {
|
||
"colorbar": {
|
||
"outlinewidth": 0,
|
||
"ticks": ""
|
||
}
|
||
},
|
||
"type": "parcoords"
|
||
}
|
||
],
|
||
"pie": [
|
||
{
|
||
"automargin": true,
|
||
"type": "pie"
|
||
}
|
||
],
|
||
"scatter": [
|
||
{
|
||
"fillpattern": {
|
||
"fillmode": "overlay",
|
||
"size": 10,
|
||
"solidity": 0.2
|
||
},
|
||
"type": "scatter"
|
||
}
|
||
],
|
||
"scatter3d": [
|
||
{
|
||
"line": {
|
||
"colorbar": {
|
||
"outlinewidth": 0,
|
||
"ticks": ""
|
||
}
|
||
},
|
||
"marker": {
|
||
"colorbar": {
|
||
"outlinewidth": 0,
|
||
"ticks": ""
|
||
}
|
||
},
|
||
"type": "scatter3d"
|
||
}
|
||
],
|
||
"scattercarpet": [
|
||
{
|
||
"marker": {
|
||
"colorbar": {
|
||
"outlinewidth": 0,
|
||
"ticks": ""
|
||
}
|
||
},
|
||
"type": "scattercarpet"
|
||
}
|
||
],
|
||
"scattergeo": [
|
||
{
|
||
"marker": {
|
||
"colorbar": {
|
||
"outlinewidth": 0,
|
||
"ticks": ""
|
||
}
|
||
},
|
||
"type": "scattergeo"
|
||
}
|
||
],
|
||
"scattergl": [
|
||
{
|
||
"marker": {
|
||
"colorbar": {
|
||
"outlinewidth": 0,
|
||
"ticks": ""
|
||
}
|
||
},
|
||
"type": "scattergl"
|
||
}
|
||
],
|
||
"scattermap": [
|
||
{
|
||
"marker": {
|
||
"colorbar": {
|
||
"outlinewidth": 0,
|
||
"ticks": ""
|
||
}
|
||
},
|
||
"type": "scattermap"
|
||
}
|
||
],
|
||
"scattermapbox": [
|
||
{
|
||
"marker": {
|
||
"colorbar": {
|
||
"outlinewidth": 0,
|
||
"ticks": ""
|
||
}
|
||
},
|
||
"type": "scattermapbox"
|
||
}
|
||
],
|
||
"scatterpolar": [
|
||
{
|
||
"marker": {
|
||
"colorbar": {
|
||
"outlinewidth": 0,
|
||
"ticks": ""
|
||
}
|
||
},
|
||
"type": "scatterpolar"
|
||
}
|
||
],
|
||
"scatterpolargl": [
|
||
{
|
||
"marker": {
|
||
"colorbar": {
|
||
"outlinewidth": 0,
|
||
"ticks": ""
|
||
}
|
||
},
|
||
"type": "scatterpolargl"
|
||
}
|
||
],
|
||
"scatterternary": [
|
||
{
|
||
"marker": {
|
||
"colorbar": {
|
||
"outlinewidth": 0,
|
||
"ticks": ""
|
||
}
|
||
},
|
||
"type": "scatterternary"
|
||
}
|
||
],
|
||
"surface": [
|
||
{
|
||
"colorbar": {
|
||
"outlinewidth": 0,
|
||
"ticks": ""
|
||
},
|
||
"colorscale": [
|
||
[
|
||
0.0,
|
||
"#0d0887"
|
||
],
|
||
[
|
||
0.1111111111111111,
|
||
"#46039f"
|
||
],
|
||
[
|
||
0.2222222222222222,
|
||
"#7201a8"
|
||
],
|
||
[
|
||
0.3333333333333333,
|
||
"#9c179e"
|
||
],
|
||
[
|
||
0.4444444444444444,
|
||
"#bd3786"
|
||
],
|
||
[
|
||
0.5555555555555556,
|
||
"#d8576b"
|
||
],
|
||
[
|
||
0.6666666666666666,
|
||
"#ed7953"
|
||
],
|
||
[
|
||
0.7777777777777778,
|
||
"#fb9f3a"
|
||
],
|
||
[
|
||
0.8888888888888888,
|
||
"#fdca26"
|
||
],
|
||
[
|
||
1.0,
|
||
"#f0f921"
|
||
]
|
||
],
|
||
"type": "surface"
|
||
}
|
||
],
|
||
"table": [
|
||
{
|
||
"cells": {
|
||
"fill": {
|
||
"color": "#EBF0F8"
|
||
},
|
||
"line": {
|
||
"color": "white"
|
||
}
|
||
},
|
||
"header": {
|
||
"fill": {
|
||
"color": "#C8D4E3"
|
||
},
|
||
"line": {
|
||
"color": "white"
|
||
}
|
||
},
|
||
"type": "table"
|
||
}
|
||
]
|
||
},
|
||
"layout": {
|
||
"annotationdefaults": {
|
||
"arrowcolor": "#2a3f5f",
|
||
"arrowhead": 0,
|
||
"arrowwidth": 1
|
||
},
|
||
"autotypenumbers": "strict",
|
||
"coloraxis": {
|
||
"colorbar": {
|
||
"outlinewidth": 0,
|
||
"ticks": ""
|
||
}
|
||
},
|
||
"colorscale": {
|
||
"diverging": [
|
||
[
|
||
0,
|
||
"#8e0152"
|
||
],
|
||
[
|
||
0.1,
|
||
"#c51b7d"
|
||
],
|
||
[
|
||
0.2,
|
||
"#de77ae"
|
||
],
|
||
[
|
||
0.3,
|
||
"#f1b6da"
|
||
],
|
||
[
|
||
0.4,
|
||
"#fde0ef"
|
||
],
|
||
[
|
||
0.5,
|
||
"#f7f7f7"
|
||
],
|
||
[
|
||
0.6,
|
||
"#e6f5d0"
|
||
],
|
||
[
|
||
0.7,
|
||
"#b8e186"
|
||
],
|
||
[
|
||
0.8,
|
||
"#7fbc41"
|
||
],
|
||
[
|
||
0.9,
|
||
"#4d9221"
|
||
],
|
||
[
|
||
1,
|
||
"#276419"
|
||
]
|
||
],
|
||
"sequential": [
|
||
[
|
||
0.0,
|
||
"#0d0887"
|
||
],
|
||
[
|
||
0.1111111111111111,
|
||
"#46039f"
|
||
],
|
||
[
|
||
0.2222222222222222,
|
||
"#7201a8"
|
||
],
|
||
[
|
||
0.3333333333333333,
|
||
"#9c179e"
|
||
],
|
||
[
|
||
0.4444444444444444,
|
||
"#bd3786"
|
||
],
|
||
[
|
||
0.5555555555555556,
|
||
"#d8576b"
|
||
],
|
||
[
|
||
0.6666666666666666,
|
||
"#ed7953"
|
||
],
|
||
[
|
||
0.7777777777777778,
|
||
"#fb9f3a"
|
||
],
|
||
[
|
||
0.8888888888888888,
|
||
"#fdca26"
|
||
],
|
||
[
|
||
1.0,
|
||
"#f0f921"
|
||
]
|
||
],
|
||
"sequentialminus": [
|
||
[
|
||
0.0,
|
||
"#0d0887"
|
||
],
|
||
[
|
||
0.1111111111111111,
|
||
"#46039f"
|
||
],
|
||
[
|
||
0.2222222222222222,
|
||
"#7201a8"
|
||
],
|
||
[
|
||
0.3333333333333333,
|
||
"#9c179e"
|
||
],
|
||
[
|
||
0.4444444444444444,
|
||
"#bd3786"
|
||
],
|
||
[
|
||
0.5555555555555556,
|
||
"#d8576b"
|
||
],
|
||
[
|
||
0.6666666666666666,
|
||
"#ed7953"
|
||
],
|
||
[
|
||
0.7777777777777778,
|
||
"#fb9f3a"
|
||
],
|
||
[
|
||
0.8888888888888888,
|
||
"#fdca26"
|
||
],
|
||
[
|
||
1.0,
|
||
"#f0f921"
|
||
]
|
||
]
|
||
},
|
||
"colorway": [
|
||
"#636efa",
|
||
"#EF553B",
|
||
"#00cc96",
|
||
"#ab63fa",
|
||
"#FFA15A",
|
||
"#19d3f3",
|
||
"#FF6692",
|
||
"#B6E880",
|
||
"#FF97FF",
|
||
"#FECB52"
|
||
],
|
||
"font": {
|
||
"color": "#2a3f5f"
|
||
},
|
||
"geo": {
|
||
"bgcolor": "white",
|
||
"lakecolor": "white",
|
||
"landcolor": "#E5ECF6",
|
||
"showlakes": true,
|
||
"showland": true,
|
||
"subunitcolor": "white"
|
||
},
|
||
"hoverlabel": {
|
||
"align": "left"
|
||
},
|
||
"hovermode": "closest",
|
||
"mapbox": {
|
||
"style": "light"
|
||
},
|
||
"paper_bgcolor": "white",
|
||
"plot_bgcolor": "#E5ECF6",
|
||
"polar": {
|
||
"angularaxis": {
|
||
"gridcolor": "white",
|
||
"linecolor": "white",
|
||
"ticks": ""
|
||
},
|
||
"bgcolor": "#E5ECF6",
|
||
"radialaxis": {
|
||
"gridcolor": "white",
|
||
"linecolor": "white",
|
||
"ticks": ""
|
||
}
|
||
},
|
||
"scene": {
|
||
"xaxis": {
|
||
"backgroundcolor": "#E5ECF6",
|
||
"gridcolor": "white",
|
||
"gridwidth": 2,
|
||
"linecolor": "white",
|
||
"showbackground": true,
|
||
"ticks": "",
|
||
"zerolinecolor": "white"
|
||
},
|
||
"yaxis": {
|
||
"backgroundcolor": "#E5ECF6",
|
||
"gridcolor": "white",
|
||
"gridwidth": 2,
|
||
"linecolor": "white",
|
||
"showbackground": true,
|
||
"ticks": "",
|
||
"zerolinecolor": "white"
|
||
},
|
||
"zaxis": {
|
||
"backgroundcolor": "#E5ECF6",
|
||
"gridcolor": "white",
|
||
"gridwidth": 2,
|
||
"linecolor": "white",
|
||
"showbackground": true,
|
||
"ticks": "",
|
||
"zerolinecolor": "white"
|
||
}
|
||
},
|
||
"shapedefaults": {
|
||
"line": {
|
||
"color": "#2a3f5f"
|
||
}
|
||
},
|
||
"ternary": {
|
||
"aaxis": {
|
||
"gridcolor": "white",
|
||
"linecolor": "white",
|
||
"ticks": ""
|
||
},
|
||
"baxis": {
|
||
"gridcolor": "white",
|
||
"linecolor": "white",
|
||
"ticks": ""
|
||
},
|
||
"bgcolor": "#E5ECF6",
|
||
"caxis": {
|
||
"gridcolor": "white",
|
||
"linecolor": "white",
|
||
"ticks": ""
|
||
}
|
||
},
|
||
"title": {
|
||
"x": 0.05
|
||
},
|
||
"xaxis": {
|
||
"automargin": true,
|
||
"gridcolor": "white",
|
||
"linecolor": "white",
|
||
"ticks": "",
|
||
"title": {
|
||
"standoff": 15
|
||
},
|
||
"zerolinecolor": "white",
|
||
"zerolinewidth": 2
|
||
},
|
||
"yaxis": {
|
||
"automargin": true,
|
||
"gridcolor": "white",
|
||
"linecolor": "white",
|
||
"ticks": "",
|
||
"title": {
|
||
"standoff": 15
|
||
},
|
||
"zerolinecolor": "white",
|
||
"zerolinewidth": 2
|
||
}
|
||
}
|
||
},
|
||
"title": {
|
||
"text": "Distribution of Match Scores"
|
||
},
|
||
"xaxis": {
|
||
"anchor": "y",
|
||
"domain": [
|
||
0.0,
|
||
1.0
|
||
],
|
||
"title": {
|
||
"text": "Fuzzy Match Score"
|
||
}
|
||
},
|
||
"yaxis": {
|
||
"anchor": "x",
|
||
"domain": [
|
||
0.0,
|
||
1.0
|
||
],
|
||
"title": {
|
||
"text": "count"
|
||
}
|
||
}
|
||
}
|
||
}
|
||
},
|
||
"metadata": {},
|
||
"output_type": "display_data"
|
||
}
|
||
],
|
||
"source": [
|
||
"# Score distribution\n",
|
||
"fig = px.histogram(\n",
|
||
" resolved_alt_data.to_pandas(),\n",
|
||
" x=\"match_score\",\n",
|
||
" nbins=20,\n",
|
||
" title=\"Distribution of Match Scores\",\n",
|
||
" labels={\"match_score\": \"Fuzzy Match Score\", \"count\": \"Count\"},\n",
|
||
")\n",
|
||
"fig.add_vline(x=75, line_dash=\"dash\", line_color=\"red\", annotation_text=\"Threshold (75)\")\n",
|
||
"fig.show()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "30c6a06f",
|
||
"metadata": {
|
||
"papermill": {
|
||
"duration": 0.002197,
|
||
"end_time": "2026-06-13T03:10:28.688810+00:00",
|
||
"exception": false,
|
||
"start_time": "2026-06-13T03:10:28.686613+00:00",
|
||
"status": "completed"
|
||
}
|
||
},
|
||
"source": [
|
||
"## Key Takeaways\n",
|
||
"\n",
|
||
"**Hierarchical matching**: Deterministic (CIK, LEI, FIGI) → Probabilistic (fuzzy strings) → embeddings, each stage handling what the previous one cannot\n",
|
||
"\n",
|
||
"**Algorithm selection**: `token_set_ratio` for word reordering, `partial_ratio` for substrings\n",
|
||
"\n",
|
||
"**Embeddings extend, not replace**: cosine similarity recovers paraphrases and tickers fuzzy misses (Amazon Web Services, msft), but renames and subsidiaries score low and non-separably — a curated alias table keyed to the security master is the real fix\n",
|
||
"\n",
|
||
"**Thresholds**: 85+ for precision, 70-80 for coverage (requires validation)\n",
|
||
"\n",
|
||
"**Libraries**: `rapidfuzz` (C-backed scorer used here), `sentence-transformers` (`all-MiniLM-L6-v2` embeddings), `polars` (data manipulation)"
|
||
]
|
||
}
|
||
],
|
||
"metadata": {
|
||
"kernelspec": {
|
||
"display_name": "Python 3 (ipykernel)",
|
||
"language": "python",
|
||
"name": "python3"
|
||
},
|
||
"language_info": {
|
||
"codemirror_mode": {
|
||
"name": "ipython",
|
||
"version": 3
|
||
},
|
||
"file_extension": ".py",
|
||
"mimetype": "text/x-python",
|
||
"name": "python",
|
||
"nbconvert_exporter": "python",
|
||
"pygments_lexer": "ipython3",
|
||
"version": "3.14.3"
|
||
},
|
||
"papermill": {
|
||
"default_parameters": {},
|
||
"duration": 8.500551,
|
||
"end_time": "2026-06-13T03:10:31.408479+00:00",
|
||
"environment_variables": {},
|
||
"exception": null,
|
||
"input_path": "04_fundamental_alternative_data/05_entity_resolution.ipynb",
|
||
"output_path": "04_fundamental_alternative_data/05_entity_resolution.ipynb",
|
||
"parameters": {},
|
||
"start_time": "2026-06-13T03:10:22.907928+00:00",
|
||
"version": "2.7.0"
|
||
},
|
||
"jupytext": {
|
||
"cell_metadata_filter": "tags,-all"
|
||
},
|
||
"ml4t_provenance": {
|
||
"source_py_blob": "06527c2eb76ed8d75fae854e0f55e0dde23b13c4",
|
||
"executed_at": "2026-06-13T03:10:31.742327+00:00",
|
||
"executor": "cpu-local",
|
||
"production": true,
|
||
"parameters": {},
|
||
"notes": "executed-state finalization batch (2026-06-13 run)"
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 5
|
||
}
|