{ "cells": [ { "cell_type": "markdown", "id": "7fb27b941602401d91542211134fc71a", "metadata": {}, "source": [ "# Hyperparameter Optimization with Native Optuna\n", "\n", "This notebook shows how to run Ludwig hyperparameter optimization using the\n", "**native Optuna executor** introduced in PR #4090.\n", "\n", "> **Note:** Requires PR #4090 to be merged, or `pip install ludwig` >= 0.14.\n", "\n", "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ludwig-ai/ludwig/blob/main/examples/hyperopt/optuna_executor.ipynb)" ] }, { "cell_type": "code", "execution_count": null, "id": "acae54e37e7d407bbb7b55eff062a284", "metadata": {}, "outputs": [], "source": [ "!pip install ludwig optuna --quiet" ] }, { "cell_type": "markdown", "id": "9a63283cbaf04dbcab1f6479b197f3a8", "metadata": {}, "source": [ "> **Dependency note:** The `optuna` executor type (`hyperopt.executor.type: optuna`) is\n", "> available from **Ludwig >= 0.14** (merged in PR #4090). Earlier versions only ship the\n", "> Ray Tune executor. To use this notebook with the development branch:\n", ">\n", "> ```bash\n", "> pip install git+https://github.com/ludwig-ai/ludwig.git@data-pipeline-hyperopt-modernization\n", "> ```" ] }, { "cell_type": "markdown", "id": "8dd0d8092fe74a7c96281538738b07e2", "metadata": {}, "source": [ "## Dataset\n", "\n", "We use the [UCI Wine Quality dataset](https://archive.ics.uci.edu/ml/datasets/wine+quality).\n", "It contains physicochemical measurements for red and white wines. We combine both files\n", "and create a **binary classification** target: `quality >= 7` → `1` (high quality),\n", "otherwise `0`." ] }, { "cell_type": "code", "execution_count": null, "id": "72eea5119410473aa328ad9291626812", "metadata": {}, "outputs": [], "source": [ "import pathlib\n", "import urllib.request\n", "\n", "import pandas as pd\n", "\n", "DATA_DIR = pathlib.Path(\"data\")\n", "DATA_DIR.mkdir(exist_ok=True)\n", "\n", "WHITE_URL = \"https://archive.ics.uci.edu/ml/machine-learning-databases/wine-quality/winequality-white.csv\"\n", "RED_URL = \"https://archive.ics.uci.edu/ml/machine-learning-databases/wine-quality/winequality-red.csv\"\n", "\n", "combined_path = DATA_DIR / \"wine_quality.csv\"\n", "\n", "if not combined_path.exists():\n", " print(\"Downloading …\")\n", " urllib.request.urlretrieve(WHITE_URL, DATA_DIR / \"winequality-white.csv\")\n", " urllib.request.urlretrieve(RED_URL, DATA_DIR / \"winequality-red.csv\")\n", "\n", " white = pd.read_csv(DATA_DIR / \"winequality-white.csv\", sep=\";\")\n", " red = pd.read_csv(DATA_DIR / \"winequality-red.csv\", sep=\";\")\n", " df = pd.concat([white, red], ignore_index=True)\n", " df[\"quality\"] = (df[\"quality\"] >= 7).astype(int)\n", " df.to_csv(combined_path, index=False)\n", "else:\n", " df = pd.read_csv(combined_path)\n", "\n", "print(f\"{len(df)} rows | {df['quality'].mean():.1%} positive (quality >= 7)\")\n", "df.head()" ] }, { "cell_type": "markdown", "id": "8edb47106e1a46a883d545849b8ab81b", "metadata": {}, "source": [ "## Define search space\n", "\n", "The `hyperopt` section of the Ludwig config specifies:\n", "\n", "- **executor** — which HPO backend to use and how many trials to run\n", "- **parameters** — the search space for each hyperparameter\n", "- **goal / metric** — what to optimise\n", "\n", "The Optuna executor supports the following `space` types:\n", "\n", "| Space | Ludwig key | Description |\n", "|---|---|---|\n", "| Log-uniform float | `loguniform` | Continuous on log scale — ideal for learning rates |\n", "| Uniform float | `float` | Continuous on linear scale — ideal for dropout |\n", "| Integer | `int` | Integer range, linear scale |\n", "| Categorical | `choice` | Discrete set of values |\n", "| Grid | `grid_search` | Exhaustive grid over a list of values |" ] }, { "cell_type": "code", "execution_count": null, "id": "10185d26023b46108eb7d9f57d49d2b3", "metadata": {}, "outputs": [], "source": [ "# Build feature list dynamically from the dataframe\n", "feature_cols = [c for c in df.columns if c != \"quality\"]\n", "\n", "config = {\n", " \"model_type\": \"ecd\",\n", " \"input_features\": [\n", " {\"name\": col, \"type\": \"number\", \"preprocessing\": {\"normalization\": \"zscore\"}} for col in feature_cols\n", " ],\n", " \"output_features\": [\n", " {\"name\": \"quality\", \"type\": \"binary\"},\n", " ],\n", " \"trainer\": {\n", " \"epochs\": 20,\n", " },\n", " # NOTE: type: optuna requires Ludwig >= 0.14 (PR #4090)\n", " \"hyperopt\": {\n", " \"executor\": {\n", " \"type\": \"optuna\",\n", " \"num_samples\": 20,\n", " \"sampler\": \"auto\", # auto, tpe, gp, cmaes, random\n", " \"pruner\": \"hyperband\", # stop bad trials early\n", " },\n", " \"parameters\": {\n", " \"trainer.learning_rate\": {\n", " \"space\": \"loguniform\",\n", " \"lower\": 1e-5,\n", " \"upper\": 1e-2,\n", " },\n", " \"trainer.batch_size\": {\n", " \"space\": \"int\",\n", " \"lower\": 16,\n", " \"upper\": 256,\n", " },\n", " \"trainer.optimizer.type\": {\n", " \"space\": \"choice\",\n", " \"categories\": [\"adam\", \"adamw\", \"radam\", \"schedule_free_adamw\"],\n", " },\n", " \"combiner.dropout\": {\n", " \"space\": \"float\",\n", " \"lower\": 0.0,\n", " \"upper\": 0.5,\n", " },\n", " },\n", " \"goal\": \"minimize\",\n", " \"metric\": \"validation.combined.loss\",\n", " \"split\": \"validation\",\n", " },\n", "}\n", "\n", "import json\n", "\n", "print(json.dumps(config[\"hyperopt\"], indent=2))" ] }, { "cell_type": "markdown", "id": "8763a12b2bbd4a93a75aff182afb95dc", "metadata": {}, "source": [ "## Run HPO\n", "\n", "`LudwigModel.hyperopt()` runs the full HPO loop and returns a list of trial results." ] }, { "cell_type": "code", "execution_count": null, "id": "7623eae2785240b9bd12b16a66d81610", "metadata": {}, "outputs": [], "source": [ "from ludwig.api import LudwigModel\n", "\n", "model = LudwigModel(config=config, logging_level=20)\n", "\n", "hyperopt_results, output_dir, _ = model.hyperopt(\n", " dataset=str(combined_path),\n", " output_directory=\"hyperopt_output\",\n", ")\n", "\n", "print(f\"\\nCompleted {len(hyperopt_results)} trials. Output: {output_dir}\")" ] }, { "cell_type": "code", "execution_count": null, "id": "7cdc8c89c7104fffa095e18ddfef8986", "metadata": {}, "outputs": [], "source": [ "# Show top-5 trials sorted by metric\n", "sorted_results = sorted(hyperopt_results, key=lambda r: r.get(\"metric_score\", float(\"inf\")))\n", "\n", "rows = []\n", "for i, r in enumerate(sorted_results[:5]):\n", " row = {\"rank\": i + 1, \"loss\": round(r.get(\"metric_score\", float(\"nan\")), 5)}\n", " row.update(r.get(\"parameters\", {}))\n", " rows.append(row)\n", "\n", "pd.DataFrame(rows)" ] }, { "cell_type": "markdown", "id": "b118ea5561624da68c537baed56e602f", "metadata": {}, "source": [ "## Sampler comparison\n", "\n", "Ludwig's Optuna executor exposes all of Optuna's built-in samplers via the `sampler` key.\n", "\n", "| Sampler | Key | Best for |\n", "|---|---|---|\n", "| **Auto** | `auto` | Default — Optuna selects the best sampler based on search space type |\n", "| **TPE** | `tpe` | General purpose; efficient with < 100 trials; the classic Optuna default |\n", "| **CMA-ES** | `cmaes` | Continuous spaces with many parameters; covariance matrix adaptation |\n", "| **GP (BoTorch)** | `gp` | Sample-efficient Bayesian optimisation; requires `pip install botorch` |\n", "| **Random** | `random` | Baseline; useful for ablations or very large search spaces |\n", "\n", "Change the sampler by editing the executor block:\n", "\n", "```python\n", "\"executor\": {\n", " \"type\": \"optuna\",\n", " \"num_samples\": 50,\n", " \"sampler\": \"tpe\", # <-- change this\n", "}\n", "```\n", "\n", "For GP, install the optional dependency first:\n", "\n", "```bash\n", "pip install botorch\n", "```" ] }, { "cell_type": "markdown", "id": "938c804e27f84196a10c8828c723f798", "metadata": {}, "source": [ "## Resumable HPO with SQLite\n", "\n", "If your HPO run is interrupted (Colab runtime reset, preempted spot instance, etc.) you can\n", "resume from where you left off by pointing Optuna at a **persistent storage** backend.\n", "\n", "Add a `storage` key to the executor:\n", "\n", "```python\n", "\"executor\": {\n", " \"type\": \"optuna\",\n", " \"num_samples\": 50,\n", " \"sampler\": \"auto\",\n", " \"storage\": \"sqlite:///optuna_results.db\", # <-- persist to disk\n", "}\n", "```\n", "\n", "Re-running `model.hyperopt()` with the same storage path will **continue the existing\n", "study** rather than starting a new one. Optuna automatically detects how many trials\n", "have already been completed and runs only the remaining ones.\n", "\n", "For distributed or cloud setups you can also use a PostgreSQL URL:\n", "\n", "```python\n", "\"storage\": \"postgresql://user:pass@host/dbname\"\n", "```" ] }, { "cell_type": "code", "execution_count": null, "id": "504fb2a444614c0babb325280ed9130a", "metadata": {}, "outputs": [], "source": [ "# Example: run with SQLite storage for resumability\n", "import copy\n", "\n", "config_resumable = copy.deepcopy(config)\n", "config_resumable[\"hyperopt\"][\"executor\"][\"storage\"] = \"sqlite:///optuna_results.db\"\n", "config_resumable[\"hyperopt\"][\"executor\"][\"num_samples\"] = 10 # fewer trials for demo\n", "\n", "print(\"Executor config:\")\n", "print(json.dumps(config_resumable[\"hyperopt\"][\"executor\"], indent=2))\n", "print(\"\\nRe-running with storage enabled — existing trials will be reused.\")\n", "\n", "model2 = LudwigModel(config=config_resumable, logging_level=20)\n", "results2, _, _ = model2.hyperopt(\n", " dataset=str(combined_path),\n", " output_directory=\"hyperopt_output_resumable\",\n", ")\n", "print(f\"Done. {len(results2)} trials.\")" ] }, { "cell_type": "markdown", "id": "59bbdb311c014d738909a11f9e486628", "metadata": {}, "source": [ "## Pruner: stop bad trials early\n", "\n", "A **pruner** monitors intermediate results reported during training and stops trials that\n", "are unlikely to beat the current best. This can dramatically reduce total compute when\n", "combined with epoch-level reporting.\n", "\n", "Ludwig's Optuna executor supports:\n", "\n", "| Pruner | Key | Description |\n", "|---|---|---|\n", "| **Hyperband** | `hyperband` | Successive halving over training steps; efficient for deep learning |\n", "| **Median** | `median` | Stops trials below the median performance at a given step |\n", "| **None** | *(omit key)* | No pruning; every trial runs to completion |\n", "\n", "```python\n", "\"executor\": {\n", " \"type\": \"optuna\",\n", " \"num_samples\": 50,\n", " \"sampler\": \"auto\",\n", " \"pruner\": \"hyperband\", # <-- add this\n", "}\n", "```\n", "\n", "Hyperband is the recommended default for neural network HPO. It requires at least\n", "`min_resource` epochs (default 1) to have completed before making pruning decisions,\n", "so short-running models (< 5 epochs) may see limited benefit." ] }, { "cell_type": "markdown", "id": "b43b363d81ae4b689946ece5c682cd59", "metadata": {}, "source": [ "## Results\n", "\n", "The cells below plot a **parallel coordinates** chart — each line is one trial,\n", "colour-coded by the validation loss. Narrow bundles indicate which regions of\n", "the search space consistently produce good results." ] }, { "cell_type": "code", "execution_count": null, "id": "8a65eabff63a45729fe45fb5ade58bdc", "metadata": {}, "outputs": [], "source": [ "# Build a dataframe of all trial results\n", "records = []\n", "for r in hyperopt_results:\n", " row = {\"loss\": r.get(\"metric_score\", float(\"nan\"))}\n", " row.update(r.get(\"parameters\", {}))\n", " records.append(row)\n", "\n", "results_df = pd.DataFrame(records)\n", "print(f\"{len(results_df)} trials\")\n", "results_df.describe()" ] }, { "cell_type": "code", "execution_count": null, "id": "c3933fab20d04ec698c2621248eb3be0", "metadata": {}, "outputs": [], "source": [ "import plotly.express as px\n", "\n", "# Map categorical optimizer to numeric for colour scale\n", "opt_map = {v: i for i, v in enumerate(results_df[\"trainer.optimizer.type\"].unique())}\n", "results_df[\"optimizer_idx\"] = results_df[\"trainer.optimizer.type\"].map(opt_map)\n", "\n", "dims = [\n", " dict(label=\"learning_rate\", values=results_df[\"trainer.learning_rate\"], type=\"log\"),\n", " dict(label=\"batch_size\", values=results_df[\"trainer.batch_size\"]),\n", " dict(\n", " label=\"optimizer\",\n", " values=results_df[\"optimizer_idx\"],\n", " tickvals=list(opt_map.values()),\n", " ticktext=list(opt_map.keys()),\n", " ),\n", " dict(label=\"dropout\", values=results_df[\"combiner.dropout\"]),\n", " dict(label=\"val loss\", values=results_df[\"loss\"]),\n", "]\n", "\n", "fig = px.parallel_coordinates(\n", " results_df,\n", " dimensions=[\"trainer.learning_rate\", \"trainer.batch_size\", \"optimizer_idx\", \"combiner.dropout\", \"loss\"],\n", " color=\"loss\",\n", " color_continuous_scale=px.colors.sequential.Viridis_r,\n", " labels={\n", " \"trainer.learning_rate\": \"learning rate\",\n", " \"trainer.batch_size\": \"batch size\",\n", " \"optimizer_idx\": \"optimizer\",\n", " \"combiner.dropout\": \"dropout\",\n", " \"loss\": \"val loss\",\n", " },\n", " title=\"HPO trials — parallel coordinates (lower loss is better)\",\n", ")\n", "fig.show()" ] }, { "cell_type": "code", "execution_count": null, "id": "4dd4641cc4064e0191573fe9c69df29b", "metadata": {}, "outputs": [], "source": [ "# Print best configuration\n", "best = sorted_results[0]\n", "print(f\"Best validation loss : {best['metric_score']:.5f}\")\n", "print(\"\\nBest hyperparameters:\")\n", "for k, v in best[\"parameters\"].items():\n", " print(f\" {k:35s} = {v}\")" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.12.0" } }, "nbformat": 4, "nbformat_minor": 5 }