wehub-resource-sync
593b94c120
pytest / Unit Tests (push) Has been cancelled
pytest / Integration (integration_tests_a) (push) Has been cancelled
pytest / Integration (integration_tests_b) (push) Has been cancelled
pytest / Integration (integration_tests_c) (push) Has been cancelled
pytest / Integration (integration_tests_d) (push) Has been cancelled
pytest / Integration (integration_tests_e) (push) Has been cancelled
pytest / Integration (integration_tests_f) (push) Has been cancelled
pytest / Integration (integration_tests_g) (push) Has been cancelled
pytest / Integration (integration_tests_h) (push) Has been cancelled
pytest / Integration (integration_tests_i) (push) Has been cancelled
pytest / Integration (integration_tests_j) (push) Has been cancelled
pytest / Distributed (distributed_a) (push) Has been cancelled
pytest / Distributed (distributed_b) (push) Has been cancelled
pytest / Distributed (distributed_c) (push) Has been cancelled
pytest / Distributed (distributed_d) (push) Has been cancelled
pytest / Distributed (distributed_e) (push) Has been cancelled
pytest / Distributed (distributed_f) (push) Has been cancelled
pytest / Minimal Install (push) Has been cancelled
pytest / Event File (push) Has been cancelled
pytest (slow) / py-slow (push) Has been cancelled
Publish JSON Schema / publish-schema (push) Has been cancelled
678 lines
26 KiB
Plaintext
678 lines
26 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "title-cell",
|
||
"metadata": {},
|
||
"source": [
|
||
"# Open-Set Recognition with MNIST: Reducing Network Agnostophobia\n",
|
||
"\n",
|
||
"[](https://colab.research.google.com/github/ludwig-ai/ludwig/blob/main/examples/open_set_recognition/open_set_mnist.ipynb)\n",
|
||
"\n",
|
||
"Standard image classifiers assign high-confidence predictions to *every* input — including images\n",
|
||
"from classes never seen during training. This is called **network agnostophobia**: the model is\n",
|
||
"incapable of saying \"I don't know.\"\n",
|
||
"\n",
|
||
"This notebook demonstrates three Ludwig models trained on MNIST digits **0–7** (known classes)\n",
|
||
"with digits **8–9** acting as the unknown/background:\n",
|
||
"\n",
|
||
"| Model | Loss | Expected behaviour |\n",
|
||
"|-------|------|--------------------|\n",
|
||
"| CE Baseline | `softmax_cross_entropy` | High confidence even on unknown digits |\n",
|
||
"| Entropic Open-Set | `entropic_open_set` | Pushes unknown confidence toward uniform |\n",
|
||
"| Objectosphere | `objectosphere` | Creates a clear logit-norm gap between known and unknown |\n",
|
||
"\n",
|
||
"**Paper:** Dhamija, Günther, Boult — *Reducing Network Agnostophobia*, NeurIPS 2018.\n",
|
||
"https://arxiv.org/abs/1811.04110"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "install-cell",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"!pip install ludwig torchvision --quiet"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "setup-header",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Setup and data preparation\n",
|
||
"\n",
|
||
"We download MNIST via `torchvision`, save each digit image as a PNG file, and build two CSVs:\n",
|
||
"\n",
|
||
"- **`train.csv`** — digits 0–7 with their true labels, plus digits 8–9 labelled as `\"background\"`\n",
|
||
"- **`test.csv`** — digits 0–7 (known) and digits 8–9 (unknown) with their true labels\n",
|
||
"\n",
|
||
"The training set teaches the open-set models what \"background\" looks like. The test set lets us\n",
|
||
"measure how confidently each model handles the unseen classes."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "data-prep",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"import csv\n",
|
||
"from pathlib import Path\n",
|
||
"\n",
|
||
"import torch\n",
|
||
"from PIL import Image\n",
|
||
"from torchvision import datasets, transforms\n",
|
||
"\n",
|
||
"# ── configuration ──────────────────────────────────────────────────────────────\n",
|
||
"DATA_DIR = Path(\"mnist_data\") # raw MNIST download\n",
|
||
"IMG_DIR = Path(\"mnist_images\") # saved PNG files\n",
|
||
"KNOWN_CLASSES = list(range(8)) # digits 0-7\n",
|
||
"UNKNOWN_CLASSES = [8, 9] # background / unknown\n",
|
||
"\n",
|
||
"# Limit samples per class so training is fast on CPU\n",
|
||
"MAX_TRAIN_KNOWN = 500 # per known class\n",
|
||
"MAX_TRAIN_UNKNOWN = 500 # per unknown class (background)\n",
|
||
"MAX_TEST_KNOWN = 200 # per known class\n",
|
||
"MAX_TEST_UNKNOWN = 200 # per unknown class\n",
|
||
"\n",
|
||
"IMG_DIR.mkdir(parents=True, exist_ok=True)\n",
|
||
"\n",
|
||
"# ── download ───────────────────────────────────────────────────────────────────\n",
|
||
"mnist_train = datasets.MNIST(str(DATA_DIR), train=True, download=True, transform=transforms.ToTensor())\n",
|
||
"mnist_test = datasets.MNIST(str(DATA_DIR), train=False, download=True, transform=transforms.ToTensor())\n",
|
||
"\n",
|
||
"print(f\"Downloaded: {len(mnist_train)} train / {len(mnist_test)} test samples\")\n",
|
||
"\n",
|
||
"\n",
|
||
"# ── helper: save image and return path ────────────────────────────────────────\n",
|
||
"def save_image(tensor: torch.Tensor, split: str, digit: int, idx: int) -> str:\n",
|
||
" \"\"\"Save a (1, H, W) float tensor as a grayscale PNG; return the file path.\"\"\"\n",
|
||
" folder = IMG_DIR / split / str(digit)\n",
|
||
" folder.mkdir(parents=True, exist_ok=True)\n",
|
||
" fpath = folder / f\"{idx:05d}.png\"\n",
|
||
" img = Image.fromarray((tensor.squeeze(0).numpy() * 255).astype(\"uint8\"), mode=\"L\")\n",
|
||
" img.save(fpath)\n",
|
||
" return str(fpath)\n",
|
||
"\n",
|
||
"\n",
|
||
"# ── build train.csv ───────────────────────────────────────────────────────────\n",
|
||
"# class counter for capping per-class samples\n",
|
||
"from collections import defaultdict\n",
|
||
"\n",
|
||
"\n",
|
||
"def build_csv(\n",
|
||
" dataset,\n",
|
||
" csv_path: str,\n",
|
||
" split: str,\n",
|
||
" known_classes,\n",
|
||
" unknown_classes,\n",
|
||
" max_known: int,\n",
|
||
" max_unknown: int,\n",
|
||
" label_unknown_as_background: bool,\n",
|
||
"):\n",
|
||
" \"\"\"\n",
|
||
" Walk *dataset*, save PNGs, write *csv_path*.\n",
|
||
"\n",
|
||
" label_unknown_as_background=True → training split (unknown → \"background\")\n",
|
||
" label_unknown_as_background=False → test split (unknown keeps true digit string)\n",
|
||
" \"\"\"\n",
|
||
" counts_known = defaultdict(int)\n",
|
||
" counts_unknown = defaultdict(int)\n",
|
||
" rows = []\n",
|
||
"\n",
|
||
" for global_idx, (img_tensor, digit) in enumerate(dataset):\n",
|
||
" digit = int(digit)\n",
|
||
" if digit in known_classes:\n",
|
||
" if counts_known[digit] >= max_known:\n",
|
||
" continue\n",
|
||
" path = save_image(img_tensor, split, digit, global_idx)\n",
|
||
" label = str(digit)\n",
|
||
" counts_known[digit] += 1\n",
|
||
" elif digit in unknown_classes:\n",
|
||
" if counts_unknown[digit] >= max_unknown:\n",
|
||
" continue\n",
|
||
" path = save_image(img_tensor, split, digit, global_idx)\n",
|
||
" label = \"background\" if label_unknown_as_background else str(digit)\n",
|
||
" counts_unknown[digit] += 1\n",
|
||
" else:\n",
|
||
" continue\n",
|
||
" rows.append({\"image_path\": path, \"label\": label})\n",
|
||
"\n",
|
||
" # stop early once all caps are met\n",
|
||
" if all(counts_known[c] >= max_known for c in known_classes) and all(\n",
|
||
" counts_unknown[c] >= max_unknown for c in unknown_classes\n",
|
||
" ):\n",
|
||
" break\n",
|
||
"\n",
|
||
" with open(csv_path, \"w\", newline=\"\") as f:\n",
|
||
" writer = csv.DictWriter(f, fieldnames=[\"image_path\", \"label\"])\n",
|
||
" writer.writeheader()\n",
|
||
" writer.writerows(rows)\n",
|
||
"\n",
|
||
" known_total = sum(counts_known.values())\n",
|
||
" unknown_total = sum(counts_unknown.values())\n",
|
||
" print(f\" {csv_path}: {known_total} known, {unknown_total} unknown/background\")\n",
|
||
" return rows\n",
|
||
"\n",
|
||
"\n",
|
||
"print(\"Building train.csv ...\")\n",
|
||
"train_rows = build_csv(\n",
|
||
" mnist_train,\n",
|
||
" \"train.csv\",\n",
|
||
" \"train\",\n",
|
||
" KNOWN_CLASSES,\n",
|
||
" UNKNOWN_CLASSES,\n",
|
||
" MAX_TRAIN_KNOWN,\n",
|
||
" MAX_TRAIN_UNKNOWN,\n",
|
||
" label_unknown_as_background=True,\n",
|
||
")\n",
|
||
"\n",
|
||
"print(\"Building test.csv ...\")\n",
|
||
"test_rows = build_csv(\n",
|
||
" mnist_test,\n",
|
||
" \"test.csv\",\n",
|
||
" \"test\",\n",
|
||
" KNOWN_CLASSES,\n",
|
||
" UNKNOWN_CLASSES,\n",
|
||
" MAX_TEST_KNOWN,\n",
|
||
" MAX_TEST_UNKNOWN,\n",
|
||
" label_unknown_as_background=False,\n",
|
||
")\n",
|
||
"\n",
|
||
"print(\"Done.\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "inspect-csv-header",
|
||
"metadata": {},
|
||
"source": [
|
||
"Let's quickly inspect what our CSVs look like."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "inspect-csv",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"import pandas as pd\n",
|
||
"\n",
|
||
"train_df = pd.read_csv(\"train.csv\")\n",
|
||
"test_df = pd.read_csv(\"test.csv\")\n",
|
||
"\n",
|
||
"print(\"train.csv label distribution:\")\n",
|
||
"print(train_df[\"label\"].value_counts().sort_index())\n",
|
||
"print()\n",
|
||
"print(\"test.csv label distribution:\")\n",
|
||
"print(test_df[\"label\"].value_counts().sort_index())\n",
|
||
"print()\n",
|
||
"print(train_df.head())"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "background-discovery-header",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Discover background class index\n",
|
||
"\n",
|
||
"Ludwig assigns integer indices to category labels based on frequency in the training data\n",
|
||
"(most frequent first, with index 0 reserved for `<UNK>`). Before using `entropic_open_set`\n",
|
||
"or `objectosphere` loss we need to know the integer index that Ludwig assigns to the\n",
|
||
"`\"background\"` label.\n",
|
||
"\n",
|
||
"The safest approach:\n",
|
||
"1. Run a short training job with the baseline config (standard cross-entropy).\n",
|
||
"2. Open `training_set_metadata.json` in the saved model directory.\n",
|
||
"3. Look up `output_features -> label -> str2idx -> \"background\"`.\n",
|
||
"\n",
|
||
"We do exactly this below — first training the baseline, then extracting the index programmatically."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "baseline-header",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Baseline: softmax cross-entropy\n",
|
||
"\n",
|
||
"The baseline model is a standard image classifier trained with softmax cross-entropy. It is\n",
|
||
"trained **only on known classes** (digits 0–7); digits 8–9 are excluded from its training data.\n",
|
||
"\n",
|
||
"At test time the baseline will happily assign high confidence to unknown digits because it has\n",
|
||
"no mechanism to express uncertainty."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "baseline-config",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"import yaml\n",
|
||
"\n",
|
||
"ENCODER = {\n",
|
||
" \"type\": \"stacked_cnn\",\n",
|
||
" \"conv_layers\": [\n",
|
||
" {\"num_filters\": 32, \"filter_size\": 3, \"pool_size\": 2, \"pool_stride\": 2},\n",
|
||
" {\"num_filters\": 64, \"filter_size\": 3, \"pool_size\": 2, \"pool_stride\": 2},\n",
|
||
" ],\n",
|
||
" \"fc_layers\": [{\"output_size\": 128, \"dropout\": 0.3}],\n",
|
||
"}\n",
|
||
"\n",
|
||
"config_baseline = {\n",
|
||
" \"model_type\": \"ecd\",\n",
|
||
" \"input_features\": [{\"name\": \"image_path\", \"type\": \"image\", \"encoder\": ENCODER}],\n",
|
||
" \"output_features\": [{\"name\": \"label\", \"type\": \"category\", \"loss\": {\"type\": \"softmax_cross_entropy\"}}],\n",
|
||
" \"trainer\": {\"epochs\": 10, \"learning_rate\": 0.001, \"batch_size\": 128},\n",
|
||
"}\n",
|
||
"\n",
|
||
"print(yaml.dump(config_baseline, default_flow_style=False))"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "baseline-train",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from ludwig.api import LudwigModel\n",
|
||
"\n",
|
||
"# The baseline is trained only on known classes — filter out the background rows\n",
|
||
"train_known_df = train_df[train_df[\"label\"] != \"background\"].copy()\n",
|
||
"print(f\"Training baseline on {len(train_known_df)} samples (known classes only)\")\n",
|
||
"\n",
|
||
"model_baseline = LudwigModel(config=config_baseline, logging_level=\"WARNING\")\n",
|
||
"_, _, output_dir_baseline = model_baseline.train(\n",
|
||
" dataset=train_known_df,\n",
|
||
" experiment_name=\"open_set_mnist\",\n",
|
||
" model_name=\"baseline\",\n",
|
||
" skip_save_processed_input=True,\n",
|
||
")\n",
|
||
"print(f\"Baseline model saved to: {output_dir_baseline}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "background-index-discovery",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Find the background class index from `training_set_metadata.json`\n",
|
||
"\n",
|
||
"Now that we have a trained model we can inspect its vocabulary. The agnostophobia models will\n",
|
||
"be trained on the *full* training set (known + background), so the background label will appear\n",
|
||
"in the vocabulary. We run a quick vocabulary fit to discover its index before training those\n",
|
||
"models."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "find-bg-index",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"import json\n",
|
||
"from pathlib import Path\n",
|
||
"\n",
|
||
"# The agnostophobia models are trained on the full dataset (including background).\n",
|
||
"# We need to know what index Ludwig will assign to \"background\" in *that* vocabulary.\n",
|
||
"# The simplest way is to train the entropic model first (or do a preprocessing run),\n",
|
||
"# but we can also use Ludwig's preprocessing API directly.\n",
|
||
"from ludwig.api import LudwigModel\n",
|
||
"\n",
|
||
"# Build a minimal config for preprocessing only\n",
|
||
"config_for_vocab = {\n",
|
||
" \"model_type\": \"ecd\",\n",
|
||
" \"input_features\": [{\"name\": \"image_path\", \"type\": \"image\", \"encoder\": ENCODER}],\n",
|
||
" \"output_features\": [{\"name\": \"label\", \"type\": \"category\", \"loss\": {\"type\": \"softmax_cross_entropy\"}}],\n",
|
||
" \"trainer\": {\"epochs\": 1, \"batch_size\": 128},\n",
|
||
"}\n",
|
||
"\n",
|
||
"vocab_model = LudwigModel(config=config_for_vocab, logging_level=\"WARNING\")\n",
|
||
"_, _, output_dir_vocab = vocab_model.train(\n",
|
||
" dataset=train_df, # full training set including background\n",
|
||
" experiment_name=\"open_set_mnist\",\n",
|
||
" model_name=\"vocab_probe\",\n",
|
||
" skip_save_processed_input=True,\n",
|
||
")\n",
|
||
"\n",
|
||
"metadata_path = Path(output_dir_vocab) / \"model\" / \"training_set_metadata.json\"\n",
|
||
"with open(metadata_path) as f:\n",
|
||
" metadata = json.load(f)\n",
|
||
"\n",
|
||
"str2idx = metadata[\"label\"][\"str2idx\"]\n",
|
||
"BACKGROUND_CLASS = str2idx[\"background\"]\n",
|
||
"\n",
|
||
"print(\"label str2idx:\")\n",
|
||
"for k, v in sorted(str2idx.items(), key=lambda x: x[1]):\n",
|
||
" marker = \" <-- background\" if k == \"background\" else \"\"\n",
|
||
" print(f\" {v:3d} : {k!r}{marker}\")\n",
|
||
"\n",
|
||
"print(f\"\\nBACKGROUND_CLASS = {BACKGROUND_CLASS}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "entropic-header",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Entropic Open-Set loss\n",
|
||
"\n",
|
||
"The Entropic Open-Set model is trained on the full dataset — known digits plus digits 8–9\n",
|
||
"labelled as `\"background\"`. For known samples the loss is standard cross-entropy. For\n",
|
||
"background samples the loss *maximises* Shannon entropy, pushing the softmax output toward\n",
|
||
"the uniform distribution.\n",
|
||
"\n",
|
||
"$$\n",
|
||
"\\mathcal{L} =\n",
|
||
" \\underbrace{-\\log p_y}_{\\text{CE on known}}\n",
|
||
" \\;+\\;\n",
|
||
" \\underbrace{\\sum_k p_k \\log p_k}_{\\text{neg-entropy on background}}\n",
|
||
"$$"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "entropic-train",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"config_entropic = {\n",
|
||
" \"model_type\": \"ecd\",\n",
|
||
" \"input_features\": [{\"name\": \"image_path\", \"type\": \"image\", \"encoder\": ENCODER}],\n",
|
||
" \"output_features\": [\n",
|
||
" {\n",
|
||
" \"name\": \"label\",\n",
|
||
" \"type\": \"category\",\n",
|
||
" \"loss\": {\n",
|
||
" \"type\": \"entropic_open_set\",\n",
|
||
" \"background_class\": BACKGROUND_CLASS,\n",
|
||
" },\n",
|
||
" }\n",
|
||
" ],\n",
|
||
" \"trainer\": {\"epochs\": 10, \"learning_rate\": 0.001, \"batch_size\": 128},\n",
|
||
"}\n",
|
||
"\n",
|
||
"print(f\"background_class = {BACKGROUND_CLASS}\")\n",
|
||
"print(yaml.dump(config_entropic, default_flow_style=False))\n",
|
||
"\n",
|
||
"model_entropic = LudwigModel(config=config_entropic, logging_level=\"WARNING\")\n",
|
||
"_, _, output_dir_entropic = model_entropic.train(\n",
|
||
" dataset=train_df,\n",
|
||
" experiment_name=\"open_set_mnist\",\n",
|
||
" model_name=\"entropic\",\n",
|
||
" skip_save_processed_input=True,\n",
|
||
")\n",
|
||
"print(f\"Entropic model saved to: {output_dir_entropic}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "objectosphere-header",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Objectosphere loss\n",
|
||
"\n",
|
||
"The Objectosphere loss extends the Entropic Open-Set loss with a logit-norm objective:\n",
|
||
"\n",
|
||
"- **Known samples**: CE + hinge `max(0, ξ – ||z||)²` pushes logit L2 norms above ξ\n",
|
||
"- **Background samples**: entropy maximisation + `ζ||z||²` suppresses logit norms toward zero\n",
|
||
"\n",
|
||
"This creates a clear norm gap between known and unknown samples, enabling a simple threshold\n",
|
||
"detector at inference time.\n",
|
||
"\n",
|
||
"$$\n",
|
||
"\\mathcal{L} =\n",
|
||
" \\underbrace{\\text{CE}(z_{\\text{known}}) + \\max(0,\\,\\xi - \\|z\\|)^2}_{\\text{known}}\n",
|
||
" \\;+\\;\n",
|
||
" \\underbrace{\\sum_k p_k \\log p_k + \\zeta\\,\\|z\\|^2}_{\\text{background}}\n",
|
||
"$$"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "objectosphere-train",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"config_objectosphere = {\n",
|
||
" \"model_type\": \"ecd\",\n",
|
||
" \"input_features\": [{\"name\": \"image_path\", \"type\": \"image\", \"encoder\": ENCODER}],\n",
|
||
" \"output_features\": [\n",
|
||
" {\n",
|
||
" \"name\": \"label\",\n",
|
||
" \"type\": \"category\",\n",
|
||
" \"loss\": {\n",
|
||
" \"type\": \"objectosphere\",\n",
|
||
" \"background_class\": BACKGROUND_CLASS,\n",
|
||
" \"xi\": 10.0,\n",
|
||
" \"zeta\": 0.1,\n",
|
||
" },\n",
|
||
" }\n",
|
||
" ],\n",
|
||
" \"trainer\": {\"epochs\": 10, \"learning_rate\": 0.001, \"batch_size\": 128},\n",
|
||
"}\n",
|
||
"\n",
|
||
"model_objectosphere = LudwigModel(config=config_objectosphere, logging_level=\"WARNING\")\n",
|
||
"_, _, output_dir_objectosphere = model_objectosphere.train(\n",
|
||
" dataset=train_df,\n",
|
||
" experiment_name=\"open_set_mnist\",\n",
|
||
" model_name=\"objectosphere\",\n",
|
||
" skip_save_processed_input=True,\n",
|
||
")\n",
|
||
"print(f\"Objectosphere model saved to: {output_dir_objectosphere}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "compare-header",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Compare: confidence on known vs unknown\n",
|
||
"\n",
|
||
"We now predict on the test set with each model and collect the **maximum softmax probability**\n",
|
||
"for each sample. A well-calibrated open-set model should:\n",
|
||
"\n",
|
||
"- Produce **high** max-prob on known digits (it is confident)\n",
|
||
"- Produce **low** max-prob on unknown digits (8 and 9) — ideally near `1/num_classes`"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "run-predictions",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"import numpy as np\n",
|
||
"\n",
|
||
"# Separate test rows into known and unknown\n",
|
||
"test_known_df = test_df[~test_df[\"label\"].isin([\"8\", \"9\"])].copy()\n",
|
||
"test_unknown_df = test_df[test_df[\"label\"].isin([\"8\", \"9\"])].copy()\n",
|
||
"\n",
|
||
"print(f\"Test known: {len(test_known_df)} samples\")\n",
|
||
"print(f\"Test unknown: {len(test_unknown_df)} samples\")\n",
|
||
"\n",
|
||
"\n",
|
||
"def get_max_probs(model, df):\n",
|
||
" \"\"\"Return an array of max softmax probabilities for each row in df.\"\"\"\n",
|
||
" preds, _ = model.predict(dataset=df, skip_save_predictions=True)\n",
|
||
" return preds[\"label_probability\"].values\n",
|
||
"\n",
|
||
"\n",
|
||
"print(\"Predicting with baseline ...\")\n",
|
||
"probs_baseline_known = get_max_probs(model_baseline, test_known_df)\n",
|
||
"probs_baseline_unknown = get_max_probs(model_baseline, test_unknown_df)\n",
|
||
"\n",
|
||
"print(\"Predicting with entropic ...\")\n",
|
||
"probs_entropic_known = get_max_probs(model_entropic, test_known_df)\n",
|
||
"probs_entropic_unknown = get_max_probs(model_entropic, test_unknown_df)\n",
|
||
"\n",
|
||
"print(\"Predicting with objectosphere ...\")\n",
|
||
"probs_obj_known = get_max_probs(model_objectosphere, test_known_df)\n",
|
||
"probs_obj_unknown = get_max_probs(model_objectosphere, test_unknown_df)\n",
|
||
"\n",
|
||
"print(\"Done.\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "plot-histograms",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"import matplotlib.pyplot as plt\n",
|
||
"\n",
|
||
"fig, axes = plt.subplots(1, 3, figsize=(15, 4), sharey=True)\n",
|
||
"bins = np.linspace(0, 1, 30)\n",
|
||
"\n",
|
||
"model_data = [\n",
|
||
" (\"CE Baseline\", probs_baseline_known, probs_baseline_unknown),\n",
|
||
" (\"Entropic Open-Set\", probs_entropic_known, probs_entropic_unknown),\n",
|
||
" (\"Objectosphere\", probs_obj_known, probs_obj_unknown),\n",
|
||
"]\n",
|
||
"\n",
|
||
"for ax, (title, known, unknown) in zip(axes, model_data):\n",
|
||
" ax.hist(known, bins=bins, alpha=0.6, color=\"steelblue\", label=f\"Known (0-7)\\nmean={known.mean():.3f}\")\n",
|
||
" ax.hist(unknown, bins=bins, alpha=0.6, color=\"orangered\", label=f\"Unknown (8-9)\\nmean={unknown.mean():.3f}\")\n",
|
||
" ax.set_title(title, fontsize=13)\n",
|
||
" ax.set_xlabel(\"Max softmax probability\")\n",
|
||
" ax.legend(fontsize=9)\n",
|
||
"\n",
|
||
"axes[0].set_ylabel(\"Count\")\n",
|
||
"fig.suptitle(\"Confidence on known vs unknown MNIST digits\", fontsize=14, y=1.02)\n",
|
||
"plt.tight_layout()\n",
|
||
"plt.savefig(\"confidence_histograms.png\", dpi=120, bbox_inches=\"tight\")\n",
|
||
"plt.show()\n",
|
||
"print(\"Saved confidence_histograms.png\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "threshold-header",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Threshold-based detection\n",
|
||
"\n",
|
||
"We can turn max softmax probability into a binary known/unknown detector by choosing a threshold\n",
|
||
"on a held-out validation set. A sample with max-prob below the threshold is flagged as unknown.\n",
|
||
"\n",
|
||
"Below we sweep thresholds and plot the True Positive Rate (TPR — fraction of unknowns correctly\n",
|
||
"flagged) against the False Positive Rate (FPR — fraction of known samples incorrectly flagged)."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "threshold-detection",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from sklearn.metrics import roc_auc_score, roc_curve\n",
|
||
"\n",
|
||
"fig, axes = plt.subplots(1, 3, figsize=(15, 4), sharey=True)\n",
|
||
"\n",
|
||
"model_data_thresh = [\n",
|
||
" (\"CE Baseline\", probs_baseline_known, probs_baseline_unknown),\n",
|
||
" (\"Entropic Open-Set\", probs_entropic_known, probs_entropic_unknown),\n",
|
||
" (\"Objectosphere\", probs_obj_known, probs_obj_unknown),\n",
|
||
"]\n",
|
||
"\n",
|
||
"for ax, (title, known, unknown) in zip(axes, model_data_thresh):\n",
|
||
" # Label: 0 = known, 1 = unknown; detector score = 1 - max_prob\n",
|
||
" y_true = np.concatenate([np.zeros(len(known)), np.ones(len(unknown))])\n",
|
||
" y_score = np.concatenate([1 - known, 1 - unknown])\n",
|
||
"\n",
|
||
" fpr, tpr, thresholds = roc_curve(y_true, y_score)\n",
|
||
" auc = roc_auc_score(y_true, y_score)\n",
|
||
"\n",
|
||
" ax.plot(fpr, tpr, lw=2, label=f\"AUC = {auc:.3f}\")\n",
|
||
" ax.plot([0, 1], [0, 1], \"--\", color=\"gray\", lw=1)\n",
|
||
" ax.set_title(title, fontsize=13)\n",
|
||
" ax.set_xlabel(\"False Positive Rate\")\n",
|
||
" ax.legend(fontsize=10)\n",
|
||
"\n",
|
||
"axes[0].set_ylabel(\"True Positive Rate\")\n",
|
||
"fig.suptitle(\"ROC curve for unknown detection (1 - max_prob threshold)\", fontsize=14, y=1.02)\n",
|
||
"plt.tight_layout()\n",
|
||
"plt.savefig(\"roc_curves.png\", dpi=120, bbox_inches=\"tight\")\n",
|
||
"plt.show()\n",
|
||
"print(\"Saved roc_curves.png\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "summary-header",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Summary table\n",
|
||
"\n",
|
||
"The table below summarises the mean maximum softmax probability on known vs unknown test samples.\n",
|
||
"A lower mean max-prob on unknowns indicates better open-set recognition."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "summary-table",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from sklearn.metrics import roc_auc_score\n",
|
||
"\n",
|
||
"rows = [\n",
|
||
" (\"CE Baseline\", probs_baseline_known, probs_baseline_unknown),\n",
|
||
" (\"Entropic Open-Set\", probs_entropic_known, probs_entropic_unknown),\n",
|
||
" (\"Objectosphere\", probs_obj_known, probs_obj_unknown),\n",
|
||
"]\n",
|
||
"\n",
|
||
"header = f\"{'Model':<22} | {'Mean max-prob (known)':>21} | {'Mean max-prob (unknown)':>23} | {'AUC (unknown det.)':>18}\"\n",
|
||
"sep = \"-\" * len(header)\n",
|
||
"print(sep)\n",
|
||
"print(header)\n",
|
||
"print(sep)\n",
|
||
"\n",
|
||
"for name, known, unknown in rows:\n",
|
||
" y_true = np.concatenate([np.zeros(len(known)), np.ones(len(unknown))])\n",
|
||
" y_score = np.concatenate([1 - known, 1 - unknown])\n",
|
||
" auc = roc_auc_score(y_true, y_score)\n",
|
||
" print(f\"{name:<22} | {known.mean():>21.4f} | {unknown.mean():>23.4f} | {auc:>18.4f}\")\n",
|
||
"\n",
|
||
"print(sep)\n",
|
||
"print()\n",
|
||
"print(\"Interpretation:\")\n",
|
||
"print(\" - CE Baseline: high confidence on unknowns — the model cannot say 'I don't know'\")\n",
|
||
"print(\" - Entropic Open-Set: lower mean max-prob on unknowns — closer to uniform distribution\")\n",
|
||
"print(\" - Objectosphere: similar to entropic; also creates a logit-norm gap (not shown here)\")"
|
||
]
|
||
}
|
||
],
|
||
"metadata": {
|
||
"colab": {
|
||
"provenance": []
|
||
},
|
||
"kernelspec": {
|
||
"display_name": "Python 3",
|
||
"language": "python",
|
||
"name": "python3"
|
||
},
|
||
"language_info": {
|
||
"name": "python",
|
||
"version": "3.12.0"
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 5
|
||
}
|