ludwig-ai--ludwig/examples/hyperopt/optuna_executor.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7fb27b941602401d91542211134fc71a",
   "metadata": {},
   "source": [
    "# Hyperparameter Optimization with Native Optuna\n",
    "\n",
    "This notebook shows how to run Ludwig hyperparameter optimization using the\n",
    "**native Optuna executor** introduced in PR #4090.\n",
    "\n",
    "> **Note:** Requires PR #4090 to be merged, or `pip install ludwig` >= 0.14.\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ludwig-ai/ludwig/blob/main/examples/hyperopt/optuna_executor.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "acae54e37e7d407bbb7b55eff062a284",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install ludwig optuna --quiet"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a63283cbaf04dbcab1f6479b197f3a8",
   "metadata": {},
   "source": [
    "> **Dependency note:** The `optuna` executor type (`hyperopt.executor.type: optuna`) is\n",
    "> available from **Ludwig >= 0.14** (merged in PR #4090). Earlier versions only ship the\n",
    "> Ray Tune executor. To use this notebook with the development branch:\n",
    ">\n",
    "> ```bash\n",
    "> pip install git+https://github.com/ludwig-ai/ludwig.git@data-pipeline-hyperopt-modernization\n",
    "> ```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8dd0d8092fe74a7c96281538738b07e2",
   "metadata": {},
   "source": [
    "## Dataset\n",
    "\n",
    "We use the [UCI Wine Quality dataset](https://archive.ics.uci.edu/ml/datasets/wine+quality).\n",
    "It contains physicochemical measurements for red and white wines. We combine both files\n",
    "and create a **binary classification** target: `quality >= 7` → `1` (high quality),\n",
    "otherwise `0`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "72eea5119410473aa328ad9291626812",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pathlib\n",
    "import urllib.request\n",
    "\n",
    "import pandas as pd\n",
    "\n",
    "DATA_DIR = pathlib.Path(\"data\")\n",
    "DATA_DIR.mkdir(exist_ok=True)\n",
    "\n",
    "WHITE_URL = \"https://archive.ics.uci.edu/ml/machine-learning-databases/wine-quality/winequality-white.csv\"\n",
    "RED_URL = \"https://archive.ics.uci.edu/ml/machine-learning-databases/wine-quality/winequality-red.csv\"\n",
    "\n",
    "combined_path = DATA_DIR / \"wine_quality.csv\"\n",
    "\n",
    "if not combined_path.exists():\n",
    "    print(\"Downloading …\")\n",
    "    urllib.request.urlretrieve(WHITE_URL, DATA_DIR / \"winequality-white.csv\")\n",
    "    urllib.request.urlretrieve(RED_URL, DATA_DIR / \"winequality-red.csv\")\n",
    "\n",
    "    white = pd.read_csv(DATA_DIR / \"winequality-white.csv\", sep=\";\")\n",
    "    red = pd.read_csv(DATA_DIR / \"winequality-red.csv\", sep=\";\")\n",
    "    df = pd.concat([white, red], ignore_index=True)\n",
    "    df[\"quality\"] = (df[\"quality\"] >= 7).astype(int)\n",
    "    df.to_csv(combined_path, index=False)\n",
    "else:\n",
    "    df = pd.read_csv(combined_path)\n",
    "\n",
    "print(f\"{len(df)} rows  |  {df['quality'].mean():.1%} positive (quality >= 7)\")\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8edb47106e1a46a883d545849b8ab81b",
   "metadata": {},
   "source": [
    "## Define search space\n",
    "\n",
    "The `hyperopt` section of the Ludwig config specifies:\n",
    "\n",
    "- **executor** — which HPO backend to use and how many trials to run\n",
    "- **parameters** — the search space for each hyperparameter\n",
    "- **goal / metric** — what to optimise\n",
    "\n",
    "The Optuna executor supports the following `space` types:\n",
    "\n",
    "| Space | Ludwig key | Description |\n",
    "|---|---|---|\n",
    "| Log-uniform float | `loguniform` | Continuous on log scale — ideal for learning rates |\n",
    "| Uniform float | `float` | Continuous on linear scale — ideal for dropout |\n",
    "| Integer | `int` | Integer range, linear scale |\n",
    "| Categorical | `choice` | Discrete set of values |\n",
    "| Grid | `grid_search` | Exhaustive grid over a list of values |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "10185d26023b46108eb7d9f57d49d2b3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Build feature list dynamically from the dataframe\n",
    "feature_cols = [c for c in df.columns if c != \"quality\"]\n",
    "\n",
    "config = {\n",
    "    \"model_type\": \"ecd\",\n",
    "    \"input_features\": [\n",
    "        {\"name\": col, \"type\": \"number\", \"preprocessing\": {\"normalization\": \"zscore\"}} for col in feature_cols\n",
    "    ],\n",
    "    \"output_features\": [\n",
    "        {\"name\": \"quality\", \"type\": \"binary\"},\n",
    "    ],\n",
    "    \"trainer\": {\n",
    "        \"epochs\": 20,\n",
    "    },\n",
    "    # NOTE: type: optuna requires Ludwig >= 0.14 (PR #4090)\n",
    "    \"hyperopt\": {\n",
    "        \"executor\": {\n",
    "            \"type\": \"optuna\",\n",
    "            \"num_samples\": 20,\n",
    "            \"sampler\": \"auto\",  # auto, tpe, gp, cmaes, random\n",
    "            \"pruner\": \"hyperband\",  # stop bad trials early\n",
    "        },\n",
    "        \"parameters\": {\n",
    "            \"trainer.learning_rate\": {\n",
    "                \"space\": \"loguniform\",\n",
    "                \"lower\": 1e-5,\n",
    "                \"upper\": 1e-2,\n",
    "            },\n",
    "            \"trainer.batch_size\": {\n",
    "                \"space\": \"int\",\n",
    "                \"lower\": 16,\n",
    "                \"upper\": 256,\n",
    "            },\n",
    "            \"trainer.optimizer.type\": {\n",
    "                \"space\": \"choice\",\n",
    "                \"categories\": [\"adam\", \"adamw\", \"radam\", \"schedule_free_adamw\"],\n",
    "            },\n",
    "            \"combiner.dropout\": {\n",
    "                \"space\": \"float\",\n",
    "                \"lower\": 0.0,\n",
    "                \"upper\": 0.5,\n",
    "            },\n",
    "        },\n",
    "        \"goal\": \"minimize\",\n",
    "        \"metric\": \"validation.combined.loss\",\n",
    "        \"split\": \"validation\",\n",
    "    },\n",
    "}\n",
    "\n",
    "import json\n",
    "\n",
    "print(json.dumps(config[\"hyperopt\"], indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8763a12b2bbd4a93a75aff182afb95dc",
   "metadata": {},
   "source": [
    "## Run HPO\n",
    "\n",
    "`LudwigModel.hyperopt()` runs the full HPO loop and returns a list of trial results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7623eae2785240b9bd12b16a66d81610",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ludwig.api import LudwigModel\n",
    "\n",
    "model = LudwigModel(config=config, logging_level=20)\n",
    "\n",
    "hyperopt_results, output_dir, _ = model.hyperopt(\n",
    "    dataset=str(combined_path),\n",
    "    output_directory=\"hyperopt_output\",\n",
    ")\n",
    "\n",
    "print(f\"\\nCompleted {len(hyperopt_results)} trials.  Output: {output_dir}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7cdc8c89c7104fffa095e18ddfef8986",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Show top-5 trials sorted by metric\n",
    "sorted_results = sorted(hyperopt_results, key=lambda r: r.get(\"metric_score\", float(\"inf\")))\n",
    "\n",
    "rows = []\n",
    "for i, r in enumerate(sorted_results[:5]):\n",
    "    row = {\"rank\": i + 1, \"loss\": round(r.get(\"metric_score\", float(\"nan\")), 5)}\n",
    "    row.update(r.get(\"parameters\", {}))\n",
    "    rows.append(row)\n",
    "\n",
    "pd.DataFrame(rows)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b118ea5561624da68c537baed56e602f",
   "metadata": {},
   "source": [
    "## Sampler comparison\n",
    "\n",
    "Ludwig's Optuna executor exposes all of Optuna's built-in samplers via the `sampler` key.\n",
    "\n",
    "| Sampler | Key | Best for |\n",
    "|---|---|---|\n",
    "| **Auto** | `auto` | Default — Optuna selects the best sampler based on search space type |\n",
    "| **TPE** | `tpe` | General purpose; efficient with < 100 trials; the classic Optuna default |\n",
    "| **CMA-ES** | `cmaes` | Continuous spaces with many parameters; covariance matrix adaptation |\n",
    "| **GP (BoTorch)** | `gp` | Sample-efficient Bayesian optimisation; requires `pip install botorch` |\n",
    "| **Random** | `random` | Baseline; useful for ablations or very large search spaces |\n",
    "\n",
    "Change the sampler by editing the executor block:\n",
    "\n",
    "```python\n",
    "\"executor\": {\n",
    "    \"type\": \"optuna\",\n",
    "    \"num_samples\": 50,\n",
    "    \"sampler\": \"tpe\",   # <-- change this\n",
    "}\n",
    "```\n",
    "\n",
    "For GP, install the optional dependency first:\n",
    "\n",
    "```bash\n",
    "pip install botorch\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "938c804e27f84196a10c8828c723f798",
   "metadata": {},
   "source": [
    "## Resumable HPO with SQLite\n",
    "\n",
    "If your HPO run is interrupted (Colab runtime reset, preempted spot instance, etc.) you can\n",
    "resume from where you left off by pointing Optuna at a **persistent storage** backend.\n",
    "\n",
    "Add a `storage` key to the executor:\n",
    "\n",
    "```python\n",
    "\"executor\": {\n",
    "    \"type\": \"optuna\",\n",
    "    \"num_samples\": 50,\n",
    "    \"sampler\": \"auto\",\n",
    "    \"storage\": \"sqlite:///optuna_results.db\",  # <-- persist to disk\n",
    "}\n",
    "```\n",
    "\n",
    "Re-running `model.hyperopt()` with the same storage path will **continue the existing\n",
    "study** rather than starting a new one. Optuna automatically detects how many trials\n",
    "have already been completed and runs only the remaining ones.\n",
    "\n",
    "For distributed or cloud setups you can also use a PostgreSQL URL:\n",
    "\n",
    "```python\n",
    "\"storage\": \"postgresql://user:pass@host/dbname\"\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "504fb2a444614c0babb325280ed9130a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Example: run with SQLite storage for resumability\n",
    "import copy\n",
    "\n",
    "config_resumable = copy.deepcopy(config)\n",
    "config_resumable[\"hyperopt\"][\"executor\"][\"storage\"] = \"sqlite:///optuna_results.db\"\n",
    "config_resumable[\"hyperopt\"][\"executor\"][\"num_samples\"] = 10  # fewer trials for demo\n",
    "\n",
    "print(\"Executor config:\")\n",
    "print(json.dumps(config_resumable[\"hyperopt\"][\"executor\"], indent=2))\n",
    "print(\"\\nRe-running with storage enabled — existing trials will be reused.\")\n",
    "\n",
    "model2 = LudwigModel(config=config_resumable, logging_level=20)\n",
    "results2, _, _ = model2.hyperopt(\n",
    "    dataset=str(combined_path),\n",
    "    output_directory=\"hyperopt_output_resumable\",\n",
    ")\n",
    "print(f\"Done. {len(results2)} trials.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59bbdb311c014d738909a11f9e486628",
   "metadata": {},
   "source": [
    "## Pruner: stop bad trials early\n",
    "\n",
    "A **pruner** monitors intermediate results reported during training and stops trials that\n",
    "are unlikely to beat the current best. This can dramatically reduce total compute when\n",
    "combined with epoch-level reporting.\n",
    "\n",
    "Ludwig's Optuna executor supports:\n",
    "\n",
    "| Pruner | Key | Description |\n",
    "|---|---|---|\n",
    "| **Hyperband** | `hyperband` | Successive halving over training steps; efficient for deep learning |\n",
    "| **Median** | `median` | Stops trials below the median performance at a given step |\n",
    "| **None** | *(omit key)* | No pruning; every trial runs to completion |\n",
    "\n",
    "```python\n",
    "\"executor\": {\n",
    "    \"type\": \"optuna\",\n",
    "    \"num_samples\": 50,\n",
    "    \"sampler\": \"auto\",\n",
    "    \"pruner\": \"hyperband\",  # <-- add this\n",
    "}\n",
    "```\n",
    "\n",
    "Hyperband is the recommended default for neural network HPO. It requires at least\n",
    "`min_resource` epochs (default 1) to have completed before making pruning decisions,\n",
    "so short-running models (< 5 epochs) may see limited benefit."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b43b363d81ae4b689946ece5c682cd59",
   "metadata": {},
   "source": [
    "## Results\n",
    "\n",
    "The cells below plot a **parallel coordinates** chart — each line is one trial,\n",
    "colour-coded by the validation loss. Narrow bundles indicate which regions of\n",
    "the search space consistently produce good results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8a65eabff63a45729fe45fb5ade58bdc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Build a dataframe of all trial results\n",
    "records = []\n",
    "for r in hyperopt_results:\n",
    "    row = {\"loss\": r.get(\"metric_score\", float(\"nan\"))}\n",
    "    row.update(r.get(\"parameters\", {}))\n",
    "    records.append(row)\n",
    "\n",
    "results_df = pd.DataFrame(records)\n",
    "print(f\"{len(results_df)} trials\")\n",
    "results_df.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c3933fab20d04ec698c2621248eb3be0",
   "metadata": {},
   "outputs": [],
   "source": [
    "import plotly.express as px\n",
    "\n",
    "# Map categorical optimizer to numeric for colour scale\n",
    "opt_map = {v: i for i, v in enumerate(results_df[\"trainer.optimizer.type\"].unique())}\n",
    "results_df[\"optimizer_idx\"] = results_df[\"trainer.optimizer.type\"].map(opt_map)\n",
    "\n",
    "dims = [\n",
    "    dict(label=\"learning_rate\", values=results_df[\"trainer.learning_rate\"], type=\"log\"),\n",
    "    dict(label=\"batch_size\", values=results_df[\"trainer.batch_size\"]),\n",
    "    dict(\n",
    "        label=\"optimizer\",\n",
    "        values=results_df[\"optimizer_idx\"],\n",
    "        tickvals=list(opt_map.values()),\n",
    "        ticktext=list(opt_map.keys()),\n",
    "    ),\n",
    "    dict(label=\"dropout\", values=results_df[\"combiner.dropout\"]),\n",
    "    dict(label=\"val loss\", values=results_df[\"loss\"]),\n",
    "]\n",
    "\n",
    "fig = px.parallel_coordinates(\n",
    "    results_df,\n",
    "    dimensions=[\"trainer.learning_rate\", \"trainer.batch_size\", \"optimizer_idx\", \"combiner.dropout\", \"loss\"],\n",
    "    color=\"loss\",\n",
    "    color_continuous_scale=px.colors.sequential.Viridis_r,\n",
    "    labels={\n",
    "        \"trainer.learning_rate\": \"learning rate\",\n",
    "        \"trainer.batch_size\": \"batch size\",\n",
    "        \"optimizer_idx\": \"optimizer\",\n",
    "        \"combiner.dropout\": \"dropout\",\n",
    "        \"loss\": \"val loss\",\n",
    "    },\n",
    "    title=\"HPO trials — parallel coordinates (lower loss is better)\",\n",
    ")\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4dd4641cc4064e0191573fe9c69df29b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print best configuration\n",
    "best = sorted_results[0]\n",
    "print(f\"Best validation loss : {best['metric_score']:.5f}\")\n",
    "print(\"\\nBest hyperparameters:\")\n",
    "for k, v in best[\"parameters\"].items():\n",
    "    print(f\"  {k:35s} = {v}\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.12.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}