chore: import upstream snapshot with attribution

.actrc

@@ -0,0 +1 @@
+-P ubuntu-latest=ludwigai/ludwig-ray

.claude/commands/ludwig_review.md

@@ -0,0 +1,70 @@
+# Ludwig Codebase Review
+
+Perform a thorough, opinionated code review of the Ludwig codebase (or a specified subsystem if an argument is given).
+
+## Scope
+
+If $ARGUMENTS is provided, scope the review to that subsystem or file pattern (e.g. `ludwig/features/`, `data pipeline`, `ray backend`).
+Otherwise review the entire codebase.
+
+## Review Dimensions
+
+Evaluate each area across ALL of the following axes:
+
+### Technical axes
+
+- **Code smells**: long methods, god objects, feature envy, primitive obsession, data clumps, shotgun surgery, dead code
+- **Duplication**: copy-paste logic, structural duplication, near-duplicate classes that should share a base
+- **Abstraction level**: too low (leaking internals), too high (over-engineered), mismatched levels within a single function
+- **Naming**: violate "naming things" rules — misleading names, abbreviations, overly generic names (`utils`, `helper`, `Manager`), names that lie about what a thing does, names that describe implementation not intent
+- **Type hints**: missing, incomplete, `Any`-abuse, wrong (e.g. `dict` where `dict[str, float]` is knowable)
+- **Docstrings**: missing on public API, wrong (describe what not why), stale (describe removed behavior)
+- **Test coverage**: untested public surface, tests that only test the happy path, tests that mock away the thing being tested, missing edge cases
+- **Performance**: unnecessary copies, redundant I/O, blocking the event loop, O(N²) in disguise, missing caching
+- **Consistency**: same concept named differently in different files, different patterns for the same operation, inconsistent error handling styles
+
+### Persona axes
+
+Rate severity from each perspective and explain why it matters to that audience:
+
+- **ML Engineer** (building production pipelines): Does this cause silent failures? Surprise OOMs? Hard-to-debug errors? Bad default choices?
+- **ML Researcher** (running experiments): Is the config surface clear? Can they reproduce results? Do names match paper terminology? Is the API discoverable?
+- **Open Source Contributor** (first PR): Is the code navigable? Is there a clear pattern to follow? Are there unexplained magic constants? Is test setup obvious?
+- **Social Media ML Reader** (HN/Reddit/X): Would they call this "spaghetti"? Is there obvious NIH syndrome? Would they praise the architecture or cringe at it?
+
+## Output Format
+
+Structure the review as:
+
+### Executive Summary
+
+2-3 sentences on overall health and the single most important thing to fix.
+
+### Critical Issues (must fix)
+
+Numbered list. Each entry: file:line_range, what's wrong, why it matters, concrete fix.
+
+### Major Issues (should fix)
+
+Same format. Things that hurt quality but aren't blocking.
+
+### Minor Issues (nice to fix)
+
+Grouped by category (naming, type hints, docstrings, etc.).
+
+### Persona Verdicts
+
+One paragraph per persona with their honest take.
+
+### Improvement Plan
+
+Ordered list of PRs/tasks to address everything, with rough size estimate (S/M/L/XL).
+
+## Instructions
+
+- Be specific: always cite file paths and line numbers (or ranges)
+- Be opinionated: don't hedge with "consider maybe possibly"
+- Don't praise things that are merely adequate
+- Distinguish between subjective style and objective bugs
+- Focus on patterns, not one-off issues — if the same problem appears in 10 files, name the pattern once and give 3 examples
+- Use the Explore subagent for broad searches, then Read for deep dives on critical files

.claude/scheduled_tasks.lock

@@ -0,0 +1 @@
+{"sessionId":"733fdee3-4ca3-414d-9f48-1cfce3f2259d","pid":132110,"procStart":"14515772","acquiredAt":1779640200070}

.claude/settings.json

@@ -0,0 +1,5 @@
+{
+  "enabledPlugins": {
+    "frontend-design@claude-plugins-official": true
+  }
+}

.deepsource.toml

@@ -0,0 +1,10 @@
+version = 1
+
+test_patterns = [
+  "tests/**"
+]
+
+[[analyzers]]
+name = "python"
+enabled = true
+runtime_version = "3.x.x"

.devcontainer/Dockerfile

@@ -0,0 +1,24 @@
+FROM python:3.12-slim
+
+RUN apt-get update && export DEBIAN_FRONTEND=noninteractive \
+    && apt-get install -y --no-install-recommends \
+        git \
+        build-essential \
+        curl \
+        libsndfile1 \
+        ffmpeg \
+        sox \
+    && apt-get clean && rm -rf /var/lib/apt/lists/*
+
+# Create non-root user
+ARG USERNAME=vscode
+ARG USER_UID=1000
+ARG USER_GID=$USER_UID
+RUN groupadd --gid $USER_GID $USERNAME \
+    && useradd --uid $USER_UID --gid $USER_GID -m $USERNAME
+
+USER $USERNAME
+ENV PATH="/home/$USERNAME/.local/bin:$PATH"
+
+# Pre-install pip tools so editable install is fast
+RUN pip install --user --no-cache-dir --upgrade pip setuptools wheel

.devcontainer/devcontainer.json

@@ -0,0 +1,24 @@
+{
+	"name": "Ludwig Dev",
+	"build": {
+		"dockerfile": "Dockerfile",
+		"context": ".."
+	},
+	"customizations": {
+		"vscode": {
+			"settings": {
+				"python.defaultInterpreterPath": "/usr/local/bin/python"
+			},
+			"extensions": [
+				"ms-python.python",
+				"ms-python.vscode-pylance",
+				"charliermarsh.ruff"
+			]
+		}
+	},
+	"postCreateCommand": "pip install --user -e '.[test]'",
+	"remoteUser": "vscode",
+	"features": {
+		"ghcr.io/devcontainers/features/git:1": {}
+	}
+}

.flake8

@@ -0,0 +1,20 @@
+[flake8]
+max-line-length = 120
+exclude =
+    .tox,
+    *.egg,
+    *_pb2.py,
+    build,
+    temp
+
+select = E,W,F
+doctests = True
+verbose = 2
+format = pylint
+# E731: Do not assign a lambda expression, use a def
+# W503: Line break occurred before a binary operator
+# E203: whitespace before ':'
+ignore =
+    E731,
+    W503,
+    E203

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,37 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: ''
+assignees: ''
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**To Reproduce**
+Steps to reproduce the behavior:
+
+1. Go to '...'
+1. Click on '....'
+1. Scroll down to '....'
+1. See error
+
+Please provide code, yaml config file and a sample of data in order to entirely reproduce the issue.
+Issues that are not reproducible will be ignored.
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Screenshots**
+If applicable, add screenshots to help explain your problem.
+
+**Environment (please complete the following information):**
+
+- OS: [e.g. iOS]
+- Version [e.g. 22]
+- Python version
+- Ludwig version
+
+**Additional context**
+Add any other context about the problem here.

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,22 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+title: ''
+labels: ''
+assignees: ''
+---
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the use case**
+A clear and concise description of what the use case for this feature is.
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Describe alternatives you've considered**
+A clear and concise description of any alternative solutions or features you've considered.
+
+**Additional context**
+Add any other context or screenshots about the feature request here.

.github/pull_request_template.md

@@ -0,0 +1,15 @@
+# Code Pull Requests
+
+Please provide the following:
+
+- a clear explanation of what your code does
+- if applicable, a reference to an issue
+- a reproducible test for your PR (code, config and data sample)
+
+# Documentation Pull Requests
+
+Note that the documentation HTML files are in `docs/` while the Markdown sources are in `mkdocs/docs`.
+
+If you are proposing a modification to the documentation you should change only the Markdown files.
+
+`api.md` is automatically generated from the docstrings in the code, so if you want to change something in that file, first modify `ludwig/api.py` docstring, then run `mkdocs/code_docs_autogen.py`, which will create `mkdocs/docs/api.md` .

.github/workflows/docker.yml

@@ -0,0 +1,86 @@
+name: docker
+
+on:
+  push:
+    tags: ["v*.*.*"]
+  workflow_dispatch:
+    inputs:
+      ludwig_version:
+        description: "PyPI version to install, e.g. 0.13.0 (leave empty to build from source)"
+        required: false
+        default: ""
+      latest:
+        description: "Also tag as :latest"
+        required: false
+        type: boolean
+        default: false
+
+jobs:
+  docker:
+    name: Build docker image ${{ matrix.docker-image }}
+    if: github.repository == 'ludwig-ai/ludwig'
+    runs-on: ubuntu-latest
+
+    # cancel in-progress runs for the same branch/tag
+    concurrency:
+      group: docker-${{ matrix.docker-image }}-${{ inputs.ludwig_version || github.head_ref || github.sha }}
+      cancel-in-progress: true
+
+    strategy:
+      fail-fast: false
+      matrix:
+        docker-image:
+          - ludwig
+          - ludwig-gpu
+          - ludwig-ray
+          - ludwig-ray-gpu
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v6
+
+      - name: Derive minor version tag
+        id: version
+        if: inputs.ludwig_version != ''
+        run: |
+          minor=$(echo "${{ inputs.ludwig_version }}" | cut -d. -f1-2)
+          echo "minor=${minor}" >> "$GITHUB_OUTPUT"
+
+      - name: Docker meta
+        id: meta
+        uses: docker/metadata-action@v6
+        with:
+          images: |
+            ludwigai/${{ matrix.docker-image }}
+          tags: |
+            # versioned dispatch: use explicit PyPI version tags
+            type=raw,value=${{ inputs.ludwig_version }},enable=${{ inputs.ludwig_version != '' }}
+            type=raw,value=${{ steps.version.outputs.minor }},enable=${{ inputs.ludwig_version != '' }}
+            type=raw,value=latest,enable=${{ inputs.ludwig_version != '' && inputs.latest }}
+            # tag push: derive tags from git ref
+            type=semver,pattern={{version}},enable=${{ inputs.ludwig_version == '' }}
+            type=semver,pattern={{major}}.{{minor}},enable=${{ inputs.ludwig_version == '' }}
+
+      - name: Set up QEMU
+        uses: docker/setup-qemu-action@v4
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v4
+
+      - name: Login to DockerHub
+        if: github.event_name != 'pull_request'
+        uses: docker/login-action@v4
+        with:
+          username: ${{ secrets.DOCKERHUB_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_TOKEN }}
+
+      - name: Build and push
+        uses: docker/build-push-action@v7
+        with:
+          context: .
+          file: ./docker/${{ matrix.docker-image }}/Dockerfile
+          push: ${{ github.event_name != 'pull_request' }}
+          tags: ${{ steps.meta.outputs.tags }}
+          labels: ${{ steps.meta.outputs.labels }}
+          build-args: |
+            LUDWIG_VERSION=${{ inputs.ludwig_version }}

.github/workflows/pytest.yml

@@ -0,0 +1,267 @@
+name: pytest
+
+on:
+  push:
+    branches: ["main", "release-*", "future-capabilities"]
+  pull_request:
+    branches: ["main", "release-*", "future-capabilities"]
+
+concurrency:
+  group: pytest-${{ github.head_ref || github.sha }}
+  cancel-in-progress: true
+
+jobs:
+  unit-tests:
+    runs-on: ubuntu-latest
+    env:
+      AWS_ACCESS_KEY_ID: ${{ secrets.LUDWIG_TESTS_AWS_ACCESS_KEY_ID }}
+      AWS_SECRET_ACCESS_KEY: ${{ secrets.LUDWIG_TESTS_AWS_SECRET_ACCESS_KEY }}
+      KAGGLE_USERNAME: ${{ secrets.KAGGLE_USERNAME }}
+      KAGGLE_KEY: ${{ secrets.KAGGLE_KEY }}
+      IS_NOT_FORK: ${{ !(github.event.pull_request.base.repo.full_name == 'ludwig-ai/ludwig' && github.event.pull_request.head.repo.fork) }}
+      UV_SYSTEM_PYTHON: "1"
+
+    name: Unit Tests
+    timeout-minutes: 60
+    steps:
+      - uses: actions/checkout@v4
+      - name: Set up Python 3.12
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Setup Linux
+        run: |
+          sudo apt-get update && sudo apt-get install -y cmake libsndfile1 libsox-dev ffmpeg
+
+      - name: uv cache
+        uses: actions/cache@v4
+        with:
+          path: ~/.cache/uv
+          key: ${{ runner.os }}-uv-unit-${{ hashFiles('pyproject.toml', '.github/workflows/pytest.yml') }}
+
+      - name: Install uv
+        run: pip install uv
+
+      - name: Install dependencies
+        run: |
+          uv pip install torch==2.12.0 torchvision==0.27.0 torchaudio==2.11.0 torchcodec --extra-index-url https://download.pytorch.org/whl/cpu
+          uv pip install '.[test]'
+          uv pip list
+
+      - name: Unit Tests
+        run: |
+          RUN_PRIVATE=$IS_NOT_FORK LUDWIG_TEST_SUITE_TIMEOUT_S=5400 pytest -v --timeout 300 --durations 100 -m "not distributed and not slow and not combinatorial and not llm" --junitxml pytest.xml tests/ludwig
+
+      - name: Regression Tests
+        run: |
+          RUN_PRIVATE=$IS_NOT_FORK LUDWIG_TEST_SUITE_TIMEOUT_S=5400 pytest -v --timeout 300 --durations 100 -m "not distributed and not slow and not combinatorial and not llm" --junitxml pytest-regression.xml tests/regression_tests
+
+      - name: Upload Test Results
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: Unit Test Results
+          path: pytest*.xml
+
+  integration-tests:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        test-markers:
+          - "integration_tests_a"
+          - "integration_tests_b"
+          - "integration_tests_c"
+          - "integration_tests_d"
+          - "integration_tests_e"
+          - "integration_tests_f"
+          - "integration_tests_g"
+          - "integration_tests_h"
+          - "integration_tests_i"
+          - "integration_tests_j"
+
+    env:
+      AWS_ACCESS_KEY_ID: ${{ secrets.LUDWIG_TESTS_AWS_ACCESS_KEY_ID }}
+      AWS_SECRET_ACCESS_KEY: ${{ secrets.LUDWIG_TESTS_AWS_SECRET_ACCESS_KEY }}
+      KAGGLE_USERNAME: ${{ secrets.KAGGLE_USERNAME }}
+      KAGGLE_KEY: ${{ secrets.KAGGLE_KEY }}
+      IS_NOT_FORK: ${{ !(github.event.pull_request.base.repo.full_name == 'ludwig-ai/ludwig' && github.event.pull_request.head.repo.fork) }}
+      MARKERS: ${{ matrix.test-markers }}
+      UV_SYSTEM_PYTHON: "1"
+      MLFLOW_ALLOW_FILE_STORE: "true"
+
+    name: Integration (${{ matrix.test-markers }})
+    services:
+      minio:
+        image: fclairamb/minio-github-actions
+        env:
+          MINIO_ACCESS_KEY: minio
+          MINIO_SECRET_KEY: minio123
+        ports:
+          - 9000:9000
+
+    timeout-minutes: 90
+    steps:
+      - uses: actions/checkout@v4
+      - name: Set up Python 3.12
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Setup Linux
+        run: |
+          sudo apt-get update && sudo apt-get install -y cmake libsndfile1 libsox-dev ffmpeg
+
+      - name: uv cache
+        uses: actions/cache@v4
+        with:
+          path: ~/.cache/uv
+          key: ${{ runner.os }}-uv-integration-${{ hashFiles('pyproject.toml', '.github/workflows/pytest.yml') }}
+
+      - name: Install dependencies
+        run: |
+          pip install uv
+          uv pip install torch==2.12.0 torchvision==0.27.0 torchaudio==2.11.0 torchcodec --extra-index-url https://download.pytorch.org/whl/cpu
+          uv pip install '.[test]'
+          uv pip list
+
+      - name: Free Disk Space
+        uses: jlumbroso/free-disk-space@main
+        with:
+          tool-cache: false
+          android: true
+          dotnet: true
+          haskell: true
+          large-packages: false
+          docker-images: true
+          swap-storage: true
+
+      - name: Integration Tests
+        run: |
+          RUN_PRIVATE=$IS_NOT_FORK LUDWIG_TEST_SUITE_TIMEOUT_S=7200 pytest -v --timeout 300 --durations 100 -m "not slow and not combinatorial and not llm and $MARKERS" --junitxml pytest.xml tests/integration_tests
+
+      - name: Upload Test Results
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: Integration Test Results (${{ matrix.test-markers }})
+          path: pytest.xml
+
+  distributed-tests:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        distributed-group:
+          - "distributed_a"
+          - "distributed_b"
+          - "distributed_c"
+          - "distributed_d"
+          - "distributed_e"
+          - "distributed_f"
+
+    env:
+      AWS_ACCESS_KEY_ID: ${{ secrets.LUDWIG_TESTS_AWS_ACCESS_KEY_ID }}
+      AWS_SECRET_ACCESS_KEY: ${{ secrets.LUDWIG_TESTS_AWS_SECRET_ACCESS_KEY }}
+      KAGGLE_USERNAME: ${{ secrets.KAGGLE_USERNAME }}
+      KAGGLE_KEY: ${{ secrets.KAGGLE_KEY }}
+      IS_NOT_FORK: ${{ !(github.event.pull_request.base.repo.full_name == 'ludwig-ai/ludwig' && github.event.pull_request.head.repo.fork) }}
+      DIST_GROUP: ${{ matrix.distributed-group }}
+      UV_SYSTEM_PYTHON: "1"
+      MLFLOW_ALLOW_FILE_STORE: "true"
+
+    name: Distributed (${{ matrix.distributed-group }})
+    services:
+      minio:
+        image: fclairamb/minio-github-actions
+        env:
+          MINIO_ACCESS_KEY: minio
+          MINIO_SECRET_KEY: minio123
+        ports:
+          - 9000:9000
+
+    timeout-minutes: 60
+    steps:
+      - uses: actions/checkout@v4
+      - name: Set up Python 3.12
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Setup Linux
+        run: |
+          sudo apt-get update && sudo apt-get install -y cmake libsndfile1 libsox-dev ffmpeg
+
+      - name: uv cache
+        uses: actions/cache@v4
+        with:
+          path: ~/.cache/uv
+          key: ${{ runner.os }}-uv-distributed-${{ hashFiles('pyproject.toml', '.github/workflows/pytest.yml') }}
+
+      - name: Install dependencies
+        run: |
+          pip install uv
+          uv pip install torch==2.12.0 torchvision==0.27.0 torchaudio==2.11.0 torchcodec --extra-index-url https://download.pytorch.org/whl/cpu
+          uv pip install '.[test]'
+          uv pip list
+
+      - name: Free Disk Space
+        uses: jlumbroso/free-disk-space@main
+        with:
+          tool-cache: false
+          android: true
+          dotnet: true
+          haskell: true
+          large-packages: false
+          docker-images: true
+          swap-storage: true
+
+      - name: Distributed Tests
+        run: |
+          RUN_PRIVATE=$IS_NOT_FORK LUDWIG_TEST_SUITE_TIMEOUT_S=5400 pytest -v --timeout 300 --durations 100 -m "distributed and not slow and not combinatorial and not llm and $DIST_GROUP" --ignore=tests/integration_tests/test_server.py --junitxml pytest.xml tests/ludwig tests/integration_tests
+
+      - name: Upload Test Results
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: Distributed Test Results (${{ matrix.distributed-group }})
+          path: pytest.xml
+
+  test-minimal-install:
+    name: Minimal Install
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    env:
+      UV_SYSTEM_PYTHON: "1"
+    steps:
+      - uses: actions/checkout@v4
+      - name: Set up Python 3.12
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Setup Linux
+        run: |
+          sudo apt-get update && sudo apt-get install -y cmake libsndfile1
+
+      - name: Install dependencies
+        run: |
+          pip install uv
+          uv pip install torch==2.12.0 torchvision==0.27.0 torchaudio==2.11.0 torchcodec --extra-index-url https://download.pytorch.org/whl/cpu
+          uv pip install .
+          uv pip list
+
+      - name: Check Install
+        run: |
+          ludwig check_install
+
+  event_file:
+    name: "Event File"
+    runs-on: ubuntu-latest
+    steps:
+      - name: Upload
+        uses: actions/upload-artifact@v4
+        with:
+          name: Event File
+          path: ${{ github.event_path }}

.github/workflows/pytest_slow.yml

@@ -0,0 +1,72 @@
+# This workflow will install Python dependencies and run all tests marked as `slow` on a single Python version.
+
+name: pytest (slow)
+
+on:
+  push:
+    branches: ["main", "release-*"]
+
+jobs:
+  slow-pytest:
+    name: py-slow
+    runs-on: ubuntu-latest
+    env:
+      # Use Minio credentials for all S3 operations in tests.
+      # PyArrow/Ray S3 clients use these env vars directly, so they must point to Minio.
+      AWS_ACCESS_KEY_ID: minio
+      AWS_SECRET_ACCESS_KEY: minio123
+      AWS_ENDPOINT_URL: http://localhost:9000
+      KAGGLE_USERNAME: ${{ secrets.KAGGLE_USERNAME }}
+      KAGGLE_KEY: ${{ secrets.KAGGLE_KEY }}
+      IS_NOT_FORK: ${{ !(github.event.pull_request.base.repo.full_name == 'ludwig-ai/ludwig' && github.event.pull_request.head.repo.fork) }}
+      MLFLOW_ALLOW_FILE_STORE: "true"
+
+    services:
+      minio:
+        image: fclairamb/minio-github-actions
+        env:
+          MINIO_ACCESS_KEY: minio
+          MINIO_SECRET_KEY: minio123
+        ports:
+          - 9000:9000
+
+    timeout-minutes: 150
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python 3.12
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Setup Linux
+        if: runner.os == 'linux'
+        run: |
+          sudo apt-get update && sudo apt-get install -y cmake libsndfile1 libsox-dev ffmpeg
+
+      - name: Install dependencies
+        run: |
+          python --version
+          pip --version
+          pip install uv
+          uv pip install --system torch==2.12.0 torchvision==0.27.0 torchaudio==2.11.0 torchcodec --extra-index-url https://download.pytorch.org/whl/cpu
+          uv pip install --system '.[test]'
+          pip list
+        shell: bash
+
+      - name: Create Minio test bucket
+        run: |
+          python -c "
+          import boto3
+          s3 = boto3.client('s3', endpoint_url='http://localhost:9000',
+                            aws_access_key_id='minio', aws_secret_access_key='minio123')
+          try:
+              s3.create_bucket(Bucket='ludwig-tests')
+          except s3.exceptions.BucketAlreadyOwnedByYou:
+              pass
+          "
+        shell: bash
+
+      - name: Tests
+        run: |
+          RUN_PRIVATE=$IS_NOT_FORK LUDWIG_TEST_SUITE_TIMEOUT_S=7200 pytest -v --timeout 600 --durations 100 -m "slow" --junitxml pytest.xml tests/integration_tests/

.github/workflows/schema.yml

@@ -0,0 +1,73 @@
+name: Publish JSON Schema
+
+on:
+  push:
+    branches: [main]
+    paths:
+      - "ludwig/schema/**"
+      - "ludwig/config_validation/**"
+      - "ludwig/constants.py"
+  release:
+    types: [published]
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  publish-schema:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install Ludwig
+        run: pip install -e ".[test]"
+
+      - name: Export schemas
+        run: |
+          mkdir -p schema-out
+          ludwig export_schema --model-type combined -o schema-out/ludwig-config.json
+          ludwig export_schema --model-type ecd -o schema-out/ludwig-config-ecd.json
+          ludwig export_schema --model-type llm -o schema-out/ludwig-config-llm.json
+
+      - name: Generate index.html
+        run: |
+          cat > schema-out/index.html << 'HTMLEOF'
+          <!DOCTYPE html>
+          <html lang="en">
+          <head>
+            <meta charset="utf-8">
+            <title>Ludwig JSON Schema</title>
+            <style>body{font-family:system-ui,sans-serif;max-width:640px;margin:2rem auto;padding:0 1rem}a{color:#0066cc}</style>
+          </head>
+          <body>
+            <h1>Ludwig JSON Schema</h1>
+            <p>JSON Schema files for <a href="https://github.com/ludwig-ai/ludwig">Ludwig</a> config validation and IDE auto-complete.</p>
+            <ul>
+              <li><a href="ludwig-config.json">ludwig-config.json</a> — Combined (ECD + LLM)</li>
+              <li><a href="ludwig-config-ecd.json">ludwig-config-ecd.json</a> — ECD only</li>
+              <li><a href="ludwig-config-llm.json">ludwig-config-llm.json</a> — LLM only</li>
+            </ul>
+            <h2>Usage</h2>
+            <p>Add to your Ludwig YAML config:</p>
+            <pre># yaml-language-server: $schema=https://ludwig-ai.github.io/schema/ludwig-config.json</pre>
+            <p>Or see <a href="https://www.schemastore.org/json/">SchemaStore</a> for automatic IDE integration.</p>
+          </body>
+          </html>
+          HTMLEOF
+
+      - name: Publish to ludwig-ai/schema
+        uses: cpina/github-action-push-to-another-repository@v1.7.2
+        env:
+          SSH_DEPLOY_KEY: ${{ secrets.SCHEMA_REPO_DEPLOY_KEY }}
+        with:
+          source-directory: schema-out
+          destination-github-username: ludwig-ai
+          destination-repository-name: schema
+          target-branch: main
+          commit-message: "Update Ludwig JSON schema from ${{ github.sha }}"

.github/workflows/test-results.yml

@@ -0,0 +1,70 @@
+name: test results
+
+# Post a clean check-run summary per test suite after the pytest workflow finishes.
+#
+# Replaces the older EnricoMi/publish-unit-test-result-action bot that posted busy
+# emoji-table PR comments accumulating across runs. dorny/test-reporter@v2 instead
+# posts a single well-formatted check run per suite with a table of passed / failed
+# tests, collapsible stack traces for failures, and source-line annotations. No
+# PR-comment spam; the result lives in the checks UI and on the "Details" page.
+#
+# Runs inside a workflow_run triggered by pytest so the downloaded JUnit XML
+# artifacts come with the write permissions needed to post check runs from
+# forked PRs (where the triggering workflow itself is read-only).
+
+on:
+  workflow_run:
+    workflows: ["pytest"]
+    types:
+      - completed
+
+permissions:
+  contents: read
+  checks: write
+  actions: read
+
+jobs:
+  test-results:
+    name: Test Results
+    runs-on: ubuntu-latest
+    if: github.event.workflow_run.conclusion != 'skipped'
+
+    steps:
+      - name: Unit Tests report
+        uses: dorny/test-reporter@v2
+        with:
+          artifact: Unit Test Results
+          name: Unit Tests
+          path: "*.xml"
+          reporter: java-junit
+          fail-on-error: false
+          list-suites: failed
+          list-tests: failed
+          max-annotations: 50
+
+      - name: Integration Tests report
+        uses: dorny/test-reporter@v2
+        with:
+          # Six matrix jobs upload artifacts named
+          # "Integration Test Results (integration_tests_a)" .. _f. Match them all with
+          # a regex and aggregate into a single report.
+          artifact: /Integration Test Results .*/
+          name: Integration Tests
+          path: "*.xml"
+          reporter: java-junit
+          fail-on-error: false
+          list-suites: failed
+          list-tests: failed
+          max-annotations: 50
+
+      - name: Distributed Tests report
+        uses: dorny/test-reporter@v2
+        with:
+          artifact: /Distributed Test Results .*/
+          name: Distributed Tests
+          path: "*.xml"
+          reporter: java-junit
+          fail-on-error: false
+          list-suites: failed
+          list-tests: failed
+          max-annotations: 50

.github/workflows/upload-pypi.yml

@@ -0,0 +1,36 @@
+name: Upload to PyPI
+
+on:
+  # Triggers the workflow when a release or draft of a release is published,
+  # or a pre-release is changed to a release
+  release:
+    types: [released]
+  # Allows you to run this workflow manually from the Actions tab
+  workflow_dispatch:
+
+jobs:
+  pypi-publish:
+    name: upload release to PyPI
+    runs-on: ubuntu-latest
+    # Specifying a GitHub environment is optional, but strongly encouraged
+    environment: release
+    permissions:
+      # IMPORTANT: this permission is mandatory for trusted publishing
+      id-token: write
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+        with:
+          submodules: "recursive"
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Build source distribution and wheel
+        run: |
+          pip install build
+          python -m build
+
+      - name: Publish package distributions to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1

.gitignore

@@ -0,0 +1,158 @@
+###################
+# ludwig specific #
+###################
+
+*.lock_preprocessing
+.plans/
+results/
+ludwig/results/
+results_*/
+ludwig_arm64/
+
+# ailabs-utils
+ailabs_util
+docker_assets
+
+# data
+mnist_data/
+profile_images/
+./profile_images/
+
+###########
+# General #
+###########
+
+# Mac stuff
+.DS_Store
+
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+env/
+env*
+build/
+develop-eggs/
+dist/
+downloads/
+./downloads/
+./dataset/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*,cover
+.hypothesis/
+
+# Data
+*.csv
+*.hdf5
+*.meta.json
+*.parquet
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# pyenv
+.python-version
+
+# celery beat schedule file
+celerybeat-schedule
+
+# SageMath parsed files
+*.sage.py
+
+# dotenv
+.env
+
+# virtualenv
+.venv
+venv*
+ENV/
+
+# Spyder project settings
+.spyderproject
+
+# Rope project settings
+.ropeproject
+
+# PyCharm
+.idea
+
+# ctags
+tags
+
+# examples
+examples/*/data/
+examples/*/visualizations/
+
+# benchmarking configs
+ludwig/benchmarking/configs/
+
+# Aim tracking
+.aim/
+
+# Comet
+.comet.config
+
+# Test-generated artifacts (image/audio features)
+*.png
+*.wav
+generated_audio/
+generated_images/
+
+# Claude Code worktrees (should never be committed)
+.claude/worktrees/

.plan

@@ -0,0 +1,315 @@
+# Ludwig Examples & Tutorials Expansion Plan
+
+## Context
+
+Two releases (0.12 → 0.13) added substantial new surface area.  The table below maps every
+significant new feature to its current documentation coverage and the example gap that needs
+filling.  The final section covers features in open PRs that are not yet merged.
+
+### Merged — what's in main right now
+
+| Feature | PR | Example gap |
+|---------|----|-------------|
+| Anomaly detection (Deep SVDD, SAD, DROCC) | #4109 | No runnable example, doc page only |
+| Image decoders: SegFormer, FPN, configurable UNet | #4100 | `semantic_segmentation/` is stale |
+| Focal, Dice, Lovász, NT-Xent, PolyLoss, Entmax losses | #4107 | Nothing |
+| LLM logits extraction, structured output, constrained decoding | #4103 | Nothing |
+| FastAPI: auto-schemas, Prometheus metrics, structured logging | #4101 | Nothing |
+| New LR schedulers: OneCycleLR, inverse_sqrt, WSD, polynomial | #4106 | Nothing |
+| New optimizers: RAdam, Adafactor, Schedule-Free AdamW, Muon, SOAP | #4105 | Nothing |
+| MLPClassifier decoder, temperature calibration, MC Dropout | #4108 | Nothing |
+| TransformerDecoder, scheduled sampling, beam search | #4102 | Nothing |
+| CLIP, DINOv2, SigLIP pretrained image encoders | standalone | Nothing |
+| RLHF alignment trainers: DPO, KTO, ORPO, GRPO | earlier | Config snippets only |
+| vLLM serving backend | #4089 | Doc section only |
+| Entropic Open-Set & Objectosphere losses | standalone | Standalone script, no doc tutorial |
+
+### Open PRs — not yet merged
+
+| Feature | PR | Branch |
+|---------|----|----|
+| LLM-driven config generation (`generate_config`) | #4092 | `future-capabilities` |
+| HyperNetworkCombiner | #4092 | `future-capabilities` |
+| Nash-MTL loss balancer | #4092 | `future-capabilities` |
+| ModelInspector, trainer mixins, registry modernization | #4091 | `api-code-quality` |
+| Native Optuna HPO executor | #4090 | `data-pipeline-hyperopt-modernization` |
+| Typed feature metadata classes | #4090 | `data-pipeline-hyperopt-modernization` |
+
+---
+
+## Tier 1 — High impact, completely uncharted (start here)
+
+### 1. Anomaly Detection Tutorial
+**Source:** PR #4109
+**Files:**
+- `examples/anomaly_detection/train_deep_svdd.py`
+- `examples/anomaly_detection/train_deep_sad.py`
+- `examples/anomaly_detection/train_drocc.py`
+- `examples/anomaly_detection/config_deep_svdd.yaml`
+- `examples/anomaly_detection/config_deep_sad.yaml`
+- `examples/anomaly_detection/config_drocc.yaml`
+- `examples/anomaly_detection/README.md`
+- `ludwig-docs/docs/examples/anomaly_detection.md`
+
+**Content:**
+- Synthetic sensor dataset (no download required), all three loss variants
+- Score histogram plots showing separation between normal and anomalous samples
+- Colab notebook: multimodal anomaly detection (tabular + text log messages)
+- Guidance on choosing the threshold (95th percentile of training scores)
+
+---
+
+### 2. Alignment / RLHF Tutorial
+**Source:** alignment trainers (DPO, KTO, ORPO, GRPO)
+**Files:**
+- `examples/alignment/train_dpo.py` — Llama-3.1-8B, public preference dataset, 4-bit QLoRA
+- `examples/alignment/train_kto.py`
+- `examples/alignment/train_orpo.py`
+- `examples/alignment/config_dpo.yaml`
+- `examples/alignment/config_kto.yaml`
+- `examples/alignment/config_orpo.yaml`
+- `examples/alignment/README.md`
+- `ludwig-docs/docs/examples/llm/alignment.md` (new)
+- `ludwig-docs/docs/user_guide/llms/finetuning.md` (expand with end-to-end walkthrough)
+
+**Content:**
+- DPO walkthrough from dataset prep to serving
+- DPO vs KTO comparison on the same preference dataset
+- When to use each method (table: data requirements, training cost, use case)
+- GRPO section with a custom reward function example (separate entry below)
+
+---
+
+### 3. vLLM Serving Tutorial
+**Source:** PR #4089 / `--backend vllm`
+**Files:**
+- `examples/serve/vllm_serving.py`
+- `examples/serve/vllm_client.py`
+- `examples/serve/prometheus_monitoring/docker-compose.yml`
+- `examples/serve/prometheus_monitoring/grafana_dashboard.json`
+- `ludwig-docs/docs/user_guide/serving.md` (expand existing)
+
+**Content:**
+- Model prep → launch → benchmark vs. default backend
+- Latency/throughput comparison chart
+- Multi-GPU launch with `--num_gpus N`
+- OpenAI-compatible endpoint usage
+- Prometheus scrape config + sample Grafana dashboard
+
+---
+
+### 4. Structured / Constrained LLM Decoding
+**Source:** PR #4103
+**Files:**
+- `examples/llm_structured_output/entity_extraction.py`
+- `examples/llm_structured_output/constrained_classification.py`
+- `examples/llm_structured_output/config_json_schema.yaml`
+- `examples/llm_structured_output/README.md`
+- `ludwig-docs/docs/user_guide/llms/structured_output.md` (new)
+
+**Content:**
+- Schema-constrained JSON generation (entity extraction → validated output)
+- Regex-constrained token generation (forced classification labels)
+- Logits extraction and custom post-processing
+- Comparison: unconstrained vs. constrained output quality
+
+---
+
+## Tier 2 — High impact, extends existing areas
+
+### 5. Open-Set Recognition — Full Tutorial
+**Source:** Entropic Open-Set & Objectosphere losses (already in `examples/open_set_recognition/`)
+**Files:**
+- `examples/open_set_recognition/train_open_set_mnist.py` (add MNIST-based Colab notebook)
+- `examples/open_set_recognition/open_set_recognition.ipynb` (Colab)
+- `ludwig-docs/docs/examples/open_set_recognition.md` (new, add to examples index)
+
+**Content:**
+- MNIST: known classes 0–7, unknown classes 8–9
+- Confidence histogram: CE vs. Entropic vs. Objectosphere
+- Full Ludwig YAML + CLI walkthrough (not just raw PyTorch)
+- Inference-time threshold selection from validation set
+- Links to the standalone PyTorch script as a "how it works" explainer
+
+---
+
+### 6. Uncertainty Quantification
+**Source:** PR #4108 (MC Dropout + temperature scaling calibration)
+**Files:**
+- `examples/uncertainty/mc_dropout.py`
+- `examples/uncertainty/temperature_calibration.py`
+- `examples/uncertainty/config_mc_dropout.yaml`
+- `examples/uncertainty/README.md`
+- `ludwig-docs/docs/user_guide/uncertainty.md` (new)
+
+**Content:**
+- Wine quality dataset (already in examples, no download)
+- Calibration curves and reliability diagrams before/after temperature scaling
+- MC Dropout: how many forward passes, variance as uncertainty proxy
+- OOD detection: uncertainty on held-out distribution shift samples
+- When to use each method
+
+---
+
+### 7. Pretrained Image Encoders — CLIP / DINOv2 / SigLIP
+**Source:** CLIP, DINOv2, SigLIP encoder integration
+**Files:**
+- `examples/image_encoders/compare_encoders.py`
+- `examples/image_encoders/few_shot_dinov2.py`
+- `examples/image_encoders/config_dinov2_linear_probe.yaml`
+- `examples/image_encoders/config_clip.yaml`
+- `examples/image_encoders/README.md`
+- `ludwig-docs/docs/configuration/features/pretrained_image_encoders.md` (new)
+
+**Content:**
+- Side-by-side: `stacked_cnn` vs. `dinov2` vs. `clip` on a small image classification task
+- Linear probing pattern: `use_pretrained: true`, `trainable: false`
+- Colab notebook: 5-shot image classification with DINOv2 (no fine-tuning)
+- Performance vs. training time trade-off table
+
+---
+
+### 8. Semantic Segmentation Refresh
+**Source:** PR #4100 (SegFormer, FPN, configurable UNet depth)
+**Files:**
+- `examples/semantic_segmentation/config_segformer.yaml` (replace stale config)
+- `examples/semantic_segmentation/config_fpn.yaml`
+- `examples/semantic_segmentation/unet_depth_sweep.py`
+- `examples/semantic_segmentation/README.md` (update)
+
+**Content:**
+- SegFormer as the new recommended default
+- FPN as the lightweight alternative
+- UNet depth sweep showing mIoU vs. parameter count
+- Update `docs/configuration/features/image_features.md` decoder section
+
+---
+
+## Tier 3 — Narrower audience, pending PR features
+
+### 9. LLM-Driven Config Generation
+**Source:** PR #4092 (`generate_config`)
+*Wait for #4092 to merge before building.*
+**Files:**
+- `examples/llm_config_generation/generate_and_train.py`
+- `ludwig-docs/docs/user_guide/llm_config_generation.md` (new)
+
+**Content:**
+- Natural language task description → validated Ludwig config → train
+- Show the validation step catching bad LLM suggestions
+- Cover both Anthropic and OpenAI backends
+- Tips for writing good task descriptions
+
+---
+
+### 10. HyperNetworkCombiner Tutorial
+**Source:** PR #4092
+*Wait for #4092 to merge before building.*
+**Files:**
+- `examples/hypernetwork/train_conditioned_model.py`
+- `examples/hypernetwork/config_hypernetwork.yaml`
+- `ludwig-docs/docs/configuration/combiners/hypernetwork.md` (new)
+
+**Content:**
+- Multimodal dataset where one modality (e.g. sensor type) should *condition* processing of others
+- Comparison: concat combiner vs. hypernetwork combiner on the same task
+- Reference to HyperFusion paper (arXiv 2403.13319)
+
+---
+
+### 11. Native Optuna HPO
+**Source:** PR #4090 (`OptunaExecutor`)
+*Wait for #4090 to merge before building.*
+**Files:**
+- `examples/hyperopt/optuna_executor.py`
+- `examples/hyperopt/config_optuna.yaml`
+- `ludwig-docs/docs/user_guide/hyperopt.md` (expand with Optuna section)
+
+**Content:**
+- Drop-in replacement for Ray Tune executor
+- Sampler comparison: TPE vs. CMA-ES vs. GPSampler
+- Persistence with SQLite for resumable HPO runs
+- Pruner (Hyperband) for early stopping of bad trials
+
+---
+
+### 12. GRPO — Reward-Based Alignment
+**Source:** GRPO trainer
+**Files:**
+- `examples/alignment/train_grpo.py`
+- `examples/alignment/config_grpo.yaml`
+- `ludwig-docs/docs/user_guide/llms/grpo.md` (new)
+
+**Content:**
+- Custom reward function (response length + factuality check)
+- When to use GRPO vs. DPO (no preference pairs needed)
+- Group normalisation: why it stabilises training vs. vanilla RL
+
+---
+
+### 13. Optimizer Guide
+**Source:** PR #4105 (Schedule-Free AdamW, Muon, Adafactor)
+**Files:**
+- `examples/optimizers/optimizer_comparison.py`
+- `ludwig-docs/docs/configuration/trainer.md` (add "Choosing an optimizer" section)
+
+**Content:**
+- Training curve comparison on a standard benchmark (MNIST or wine quality)
+- Schedule-Free AdamW: why no LR scheduler is needed
+- Muon: weight-matrix-only updates, when beneficial
+- Decision tree: Adam → AdamW → Schedule-Free → Muon/SOAP
+
+---
+
+### 14. Nash-MTL Multi-Task Loss Balancing
+**Source:** PR #4092 (Nash-MTL loss balancer)
+*Wait for #4092 to merge before building.*
+**Files:**
+- `examples/multi_task/nash_mtl.py`
+- `ludwig-docs/docs/user_guide/multi_task.md` (expand)
+
+**Content:**
+- Multi-output model (classification + regression simultaneously)
+- Comparison: fixed weights vs. FAMO vs. Nash-MTL loss balancing
+- When Nash-MTL is worth the overhead vs. simpler methods
+
+---
+
+## Execution order
+
+```
+Phase A (no GPU needed, self-contained):
+  1. Anomaly detection tutorial
+  5. Open-set recognition → full tutorial
+  6. Uncertainty quantification
+  13. Optimizer guide
+
+Phase B (needs GPU, builds on existing LLM work):
+  2. Alignment / RLHF tutorial (DPO + KTO)
+  3. vLLM serving tutorial
+  4. Structured / constrained LLM decoding
+  12. GRPO
+
+Phase C (vision, needs pretrained model weights):
+  7. CLIP / DINOv2 / SigLIP encoders
+  8. Semantic segmentation refresh
+
+Phase D (blocked on open PRs merging):
+  9.  LLM-driven config generation   [blocked on #4092]
+  10. HyperNetworkCombiner            [blocked on #4092]
+  14. Nash-MTL multi-task             [blocked on #4092]
+  11. Native Optuna HPO               [blocked on #4090]
+```
+
+---
+
+## Deliverable locations
+
+| Type | Repo | Path |
+|------|------|------|
+| Runnable scripts | `ludwig` | `examples/<topic>/` |
+| YAML configs | `ludwig` | `examples/<topic>/config*.yaml` |
+| Colab notebooks | `ludwig` | `examples/<topic>/*.ipynb` |
+| Doc tutorials | `ludwig-docs` | `docs/examples/<topic>.md` |
+| User guide pages | `ludwig-docs` | `docs/user_guide/<topic>.md` |
+| Config reference pages | `ludwig-docs` | `docs/configuration/**/<topic>.md` |

.plan-lazy-preprocessing.md

@@ -0,0 +1,320 @@
+# Ludwig Lazy / Streaming Preprocessing Plan
+
+## Problem Statement
+
+Ludwig's preprocessing pipeline is **not lazy**: before the training loop starts, it decodes and
+transforms every sample in the dataset into in-memory NumPy arrays. This causes:
+
+1. **OOM on media datasets** — 1 000 images at original resolution × their tensor size saturates
+   RAM before a single batch reaches the GPU.
+1. **Fixed preprocessing bottleneck** — transforms run single-threaded on the CPU ahead of training,
+   blocking the GPU.
+1. **No online statistics** — normalization stats (mean/std) require a full pass that materialises
+   the entire dataset.
+1. **Impossible streaming** — datasets that exceed RAM (or that arrive over a network) cannot be
+   used at all today.
+
+______________________________________________________________________
+
+## How Other Frameworks Solve This
+
+### PyTorch / torchvision
+
+`Dataset.__getitem__` is the canonical lazy primitive. Each DataLoader worker calls it
+independently, so only `batch_size × num_workers × prefetch_factor` samples are alive at once.
+Transforms (resize, normalize, augment) are composed in the Dataset constructor and run inside
+worker processes, overlapping with the GPU forward pass.
+
+Statistics (ImageNet mean/std) are pre-computed offline and shipped as constants. For custom
+datasets a single Welford accumulation scan (O(1) memory) is standard.
+
+### TensorFlow `tf.data`
+
+Declarative lazy pipeline: `.map(fn, num_parallel_calls=AUTOTUNE)` → `.batch()` →
+`.prefetch(AUTOTUNE)`. Nothing is executed until the training iterator pulls a batch.
+`.cache()` optionally persists the decoded dataset to disk after the first epoch.
+
+### HuggingFace `datasets`
+
+Arrow IPC memory-map: only the pages actually accessed are loaded from disk. Audio/image columns
+store raw bytes or file paths — decoded waveforms are produced per-sample at access time via
+`cast_column("audio", Audio())`. Passing `decode=False` gives raw bytes with zero decoding cost.
+
+### MONAI (3D medical imaging)
+
+Lazy resampling accumulates sequential spatial transforms (rotate → crop → resize) as pending
+homogeneous-matrix operations and fuses them into a single interpolation, eliminating intermediate
+materialisation and reducing quality loss.
+
+### torchaudio
+
+Transforms (`MelSpectrogram`, `Resample`, `MFCC`) are `nn.Module` subclasses — composable,
+JIT-scriptable, and GPU-movable. Applying them post-batch on GPU is 10-20× faster than per-sample
+on CPU.
+
+______________________________________________________________________
+
+## Design Principles for Ludwig's New Pipeline
+
+1. **Store paths, not arrays.** The Arrow dataset (or in-memory DataFrame) holds file paths /
+   metadata, never decoded tensors. Decode happens inside the DataLoader worker.
+1. **Transforms as `nn.Module`.** All feature-specific transforms become PyTorch modules:
+   composable, testable, batchable, GPU-ready.
+1. **Online statistics with Welford's algorithm.** Replace full-dataset decode-then-accumulate
+   with a single streaming pass that needs O(1) memory per feature.
+1. **Prefetching and parallelism.** DataLoader `num_workers > 0` + `prefetch_factor` provides
+   CPU/GPU overlap for free once decode is inside `__getitem__`.
+1. **Backward compatibility.** Existing configs that work today must continue to work. New
+   behaviour is opt-in at the feature level via `lazy: true` or automatic for audio/image features.
+
+______________________________________________________________________
+
+## Implementation Phases
+
+______________________________________________________________________
+
+### Phase 0 — Foundations (no user-facing change)
+
+**Goal:** establish shared infrastructure that later phases build on.
+
+#### 0-A. Welford online stats accumulator
+
+- Add `ludwig/data/statistics.py` with `WelfordAccumulator` (supports merge across shards).
+- API: `acc.update(x: np.ndarray)`, `acc.result() → (mean, std, min, max, count)`,
+  `acc.merge(other)`.
+- Unit-test with known distributions.
+
+#### 0-B. Feature transform protocol
+
+- Define `FeatureTransform(nn.Module)` base class in `ludwig/features/transforms.py`.
+- Existing preprocessing logic stays unchanged for now; this just establishes the interface.
+- `__call__(self, x: Tensor) → Tensor`; serialisable via `torch.save`.
+
+#### 0-C. DataLoader worker-safe file handle pool
+
+- Research whether Ludwig's existing multiprocessing setup needs `worker_init_fn` to reset
+  open file handles or RNG state (same issue as HuggingFace's `IterableDataset` shard split).
+- Document the policy; no code change yet.
+
+**Exit criteria:** `WelfordAccumulator` tests pass; `FeatureTransform` ABC exists.
+
+______________________________________________________________________
+
+### Phase 1 — Lazy Audio (highest OOM risk)
+
+**Goal:** audio columns decode inside the DataLoader worker, not during preprocessing.
+
+#### 1-A. `AudioFeature` path-only mode
+
+- After the preprocessing step, the Arrow/DataFrame column holds **file paths** (strings), not
+  decoded tensors.
+- Remove `_process_in_memory` eager decode; replace with a lightweight path-validity check.
+- The existing `read_audio_files_to_dict` path (used today) becomes a fallback for non-lazy mode.
+
+#### 1-B. `AudioDataset.__getitem__` decode
+
+- Create `ludwig/data/datasets/audio_dataset.py` (or extend `LudwigDataset`).
+- In `__getitem__`, call `torchaudio.load(path)` → resample → pad/trim to target length.
+- Apply `FeatureTransform` chain (mel spectrogram, normalization).
+
+#### 1-C. Online stats for audio normalization
+
+- Replace current two-pass (decode all → compute stats) with a streaming Welford scan:
+  `torchaudio.load` one file at a time, update accumulator, discard tensor.
+- Triggered once at `prepare_data` time; stats stored in the feature config cache.
+
+#### 1-D. Integration test
+
+- Run `pytest tests/integration_tests/test_audio_feature.py` — no OOM on the medium-size
+  audio fixtures.
+- Add a test that processes 10 000 short clips and checks RSS stays under 2 GB.
+
+**Exit criteria:** audio smoke tests pass on datasets with >10 h of audio; RSS bounded by
+`batch_size × clip_length × sr × 4 bytes`.
+
+______________________________________________________________________
+
+### Phase 2 — Lazy Image
+
+**Goal:** image columns store paths; decode + augment happens per-sample in DataLoader workers.
+
+#### 2-A. `ImageFeature` path-only mode
+
+- Preprocessing stores file paths (already partially the case for CSV/HDF5 workflows).
+- Remove upfront `_process_in_memory` full-batch decode; keep only metadata inference
+  (height, width, channels) via a single sampled read.
+
+#### 2-B. `ImageDataset.__getitem__` decode
+
+- `PIL.Image.open(path).convert("RGB")` → `torchvision.transforms` pipeline.
+- Apply `FeatureTransform` (resize, center crop, normalize).
+- Multi-channel and TIFF support via existing `read_image_from_path` helper.
+
+#### 2-C. Online stats for image normalization
+
+- Welford accumulator over pixel values per channel; one forward pass, batch-level updates
+  (read image → reshape to (H×W, C) → `acc.update`).
+- Offer `use_imagenet_stats: true` shortcut (ships precomputed constants) for the common case.
+
+#### 2-D. Augmentation support
+
+- `ImageFeature` config gains an optional `augmentation:` block (equivalent to
+  `torchvision.transforms.v2`).
+- Augmentation runs inside `__getitem__` → free via DataLoader parallelism.
+
+**Exit criteria:** image smoke tests pass; openfake and synthia complete without OOM at
+original resolution.
+
+______________________________________________________________________
+
+### Phase 3 — Lazy Text / Tabular (lower priority)
+
+**Goal:** tokenisation and numerical transforms move into the DataLoader worker.
+
+#### 3-A. Text tokenisation in `__getitem__`
+
+- Currently: tokenise all text upfront into integer id arrays → store in HDF5.
+- New: store raw strings; tokenise on-the-fly in `__getitem__`.
+- Tradeoff: tokenisation is cheap but repeated; cache tokenised output to Arrow after first epoch
+  if `cache_encoder_embeddings: true`.
+
+#### 3-B. Numerical normalisation as `nn.Module`
+
+- `NormalizationTransform(mean, std)` replaces the current in-place column mutation.
+- Stats computed once via Welford in `prepare_data`; transform applied in `__getitem__`.
+
+#### 3-C. Category embedding stays eager
+
+- Category integer encoding is O(1) per sample and zero-memory; no change needed.
+
+**Exit criteria:** existing tabular test suite passes unchanged.
+
+______________________________________________________________________
+
+### Phase 4 — Streaming Dataset Source
+
+**Goal:** support HuggingFace streaming datasets (`streaming=True`) as a first-class source,
+enabling datasets that exceed local disk space.
+
+#### 4-A. `HFStreamingDataset(IterableDataset)`
+
+- Wraps a `datasets.IterableDataset` in a PyTorch `IterableDataset`.
+- Shard splitting: `worker_init_fn` slices the stream across DataLoader workers (HF provides
+  `dataset.shard(num_shards=N, index=i)`).
+- Shuffle buffer: configurable per-dataset (small for media, large for text).
+
+#### 4-B. Online stats for streaming sources
+
+- Welford accumulator updated during the stats-scan pre-training pass (a full iteration over
+  the stream without gradient accumulation).
+- Stats cached to disk keyed by dataset id + revision + split.
+
+#### 4-C. No-shuffle streaming mode
+
+- For datasets too large even for a shuffle buffer, expose `shuffle: false` (warn user).
+
+**Exit criteria:** `AudioSet`, `VoxPopuli`, `LibriSpeech` stream through without touching disk;
+stats computed in one pass.
+
+______________________________________________________________________
+
+### Phase 5 — MONAI-style Transform Fusion (stretch goal)
+
+**Goal:** fuse adjacent spatial transforms (resize → crop → normalize) into a single tensor
+operation to eliminate intermediate materialisation.
+
+#### 5-A. Transform graph analysis
+
+- At `setup` time, walk the `FeatureTransform` chain; identify composable sequences.
+- A sequence is composable when all transforms implement `as_matrix() → Tensor[4×4]`.
+
+#### 5-B. Single-pass spatial transform
+
+- Fuse the homogeneous matrices; apply via `F.grid_sample` once.
+- Benchmark vs. sequential PIL transforms on ImageNet validation set.
+
+#### 5-C. Augmentation randomness
+
+- Random transforms (flip, rotation jitter) break deterministic fusion; keep them as
+  post-fusion `nn.Module` steps.
+
+**Exit criteria:** ≥ 15% wall-clock speedup on image-only training runs vs. Phase 2 baseline.
+
+______________________________________________________________________
+
+## Data-Flow Diagram (target state)
+
+```
+Disk / HF Hub
+    │
+    ▼
+┌──────────────────────────────────────────────────────┐
+│  prepare_data  (single process, once per dataset)    │
+│                                                      │
+│  • Build Arrow metadata table (paths, labels, ids)   │
+│  • Welford scan → normalization stats                │
+│  • Write stats to feature config cache              │
+└───────────────────────────┬──────────────────────────┘
+                            │ Arrow file / paths
+                            ▼
+┌──────────────────────────────────────────────────────┐
+│  LudwigDataset.__getitem__  (per DataLoader worker)  │
+│                                                      │
+│  • Read path from Arrow row                          │
+│  • torchaudio.load / PIL.Image.open                  │
+│  • FeatureTransform chain (resize, mel spec, norm)   │
+│  • Return sample dict of tensors                     │
+└───────────────────────────┬──────────────────────────┘
+                            │ batch of tensors (pinned RAM)
+                            ▼
+┌──────────────────────────────────────────────────────┐
+│  GPU training loop                                   │
+│  • H2D transfer (pin_memory → non-blocking)          │
+│  • Forward pass                                      │
+│  • Gradient accumulation                             │
+└──────────────────────────────────────────────────────┘
+```
+
+______________________________________________________________________
+
+## Key Files to Create / Modify
+
+| File                                           | Action                                              |
+| ---------------------------------------------- | --------------------------------------------------- |
+| `ludwig/data/statistics.py`                    | NEW — `WelfordAccumulator`                          |
+| `ludwig/features/transforms.py`                | NEW — `FeatureTransform` base + concrete transforms |
+| `ludwig/features/audio_feature.py`             | MODIFY — remove eager decode, add lazy path         |
+| `ludwig/features/image_feature.py`             | MODIFY — remove eager decode, add lazy path         |
+| `ludwig/data/datasets/base_dataset.py`         | MODIFY — add `__getitem__` decode hooks             |
+| `ludwig/data/datasets/audio_dataset.py`        | NEW (or extend base)                                |
+| `ludwig/data/datasets/image_dataset.py`        | NEW (or extend base)                                |
+| `ludwig/data/datasets/hf_streaming_dataset.py` | NEW — Phase 4                                       |
+| `tests/unit/data/test_statistics.py`           | NEW — Welford unit tests                            |
+| `tests/integration_tests/test_lazy_audio.py`   | NEW                                                 |
+| `tests/integration_tests/test_lazy_image.py`   | NEW                                                 |
+
+______________________________________________________________________
+
+## Risks and Mitigations
+
+| Risk                                                               | Mitigation                                                                                        |
+| ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------- |
+| DataLoader worker forking copies open file handles / CUDA contexts | `worker_init_fn` resets handles; delay CUDA init until after fork                                 |
+| Welford online stats diverge from current batch stats              | Unit-test Welford against numpy batch stats on synthetic data                                     |
+| Per-sample disk I/O is slower than batched HDF5 read               | Benchmark; add optional post-first-epoch disk cache for decoded tensors                           |
+| Existing HDF5 preprocessing cache invalidated                      | Introduce cache version key; fall back to eager mode on cache miss                                |
+| `IterableDataset` with shuffle buffer OOMs on large-media streams  | Default shuffle buffer scales with declared feature size; audio → 500, image → 200, text → 10 000 |
+
+______________________________________________________________________
+
+## Open Questions
+
+1. **HDF5 cache vs. Arrow:** Should the decoded-tensor cache (post-Phase 2) write Arrow or HDF5?
+   Arrow supports mmap natively; HDF5 has chunked access. Arrow preferred for new code.
+1. **Normalization stat freshness:** If the dataset changes between runs, cached Welford stats
+   are stale. Hash the dataset id + revision + split into the cache key.
+1. **Multi-output features:** When one column feeds multiple output features (e.g. image →
+   classification + detection), the decode must happen once. Ensure `LudwigDataset` returns a
+   shared tensor, not independent copies.
+1. **Tokenisation caching:** Tokenising every sample every epoch wastes CPU. Cache to Arrow after
+   the first epoch under `${cache_dir}/<dataset_hash>/tokenized/`.

.pre-commit-config.yaml

@@ -0,0 +1,61 @@
+# Apply to all files without committing:
+#   pre-commit run --all-files
+# Apply to changed files:
+#   pre-commit run
+# Update this file:
+#   pre-commit autoupdate
+# Run a specific hook:
+#   pre-commit run <hook id>
+
+ci:
+  autofix_prs: true
+  autoupdate_commit_msg: "[pre-commit.ci] pre-commit suggestions"
+  autoupdate_schedule: weekly
+
+# Pin to the system Python version (3.14 post OS upgrade; previously 3.12).
+# Note: pre-commit.ci hosted runners default to 3.14 which is now also local.
+default_language_version:
+  python: python3.14
+
+repos:
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v6.0.0
+    hooks:
+      - id: check-ast
+      - id: fix-byte-order-marker
+      - id: check-case-conflict
+      - id: check-executables-have-shebangs
+      - id: check-json
+      - id: check-toml
+      - id: check-yaml
+      - id: debug-statements
+      - id: detect-private-key
+      - id: end-of-file-fixer
+      - id: trailing-whitespace
+      - id: mixed-line-ending
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.15.14
+    hooks:
+      - id: ruff
+        name: Lint and fix (ruff)
+        args: [--fix]
+      - id: ruff-format
+        name: Format code (ruff)
+  - repo: https://github.com/asottile/blacken-docs
+    rev: 1.20.0
+    hooks:
+      - id: blacken-docs
+        args: [--line-length=120]
+        exclude: "IMPROVEMENT_PLAN.md"
+  - repo: https://github.com/hukkin/mdformat
+    rev: 1.0.0
+    hooks:
+      - id: mdformat
+        additional_dependencies:
+          - mdformat-gfm==1.0.0
+          - mdformat_frontmatter==2.0.10
+        exclude: "CHANGELOG.md|IMPROVEMENT_PLAN.md"
+  - repo: https://github.com/yoheimuta/protolint
+    rev: v0.56.4
+    hooks:
+      - id: protolint

.protolint.yaml

@@ -0,0 +1,53 @@
+# Adapted from
+# https://github.com/yoheimuta/protolint/blob/master/_example/config/.protolint.yaml
+---
+# Lint directives.
+lint:
+  # Linter files to walk.
+  files:
+    # The specific files to exclude.
+    exclude:
+      # NOTE: UNIX paths will be properly accepted by both UNIX and Windows.
+      - ../proto/invalidFileName.proto
+
+  # Linter rules.
+  # Run `protolint list` to see all available rules.
+  rules:
+    # Set the default to all linters. This option works the other way around as no_default does.
+    # If you want to enable this option, delete the comment out below and no_default.
+    # all_default: true
+
+    # The specific linters to add.
+    add:
+      - FIELD_NAMES_LOWER_SNAKE_CASE
+      - MESSAGE_NAMES_UPPER_CAMEL_CASE
+      - MAX_LINE_LENGTH
+      - INDENT
+      - FIELD_NAMES_EXCLUDE_PREPOSITIONS
+      - FILE_NAMES_LOWER_SNAKE_CASE
+      - IMPORTS_SORTED
+      - PACKAGE_NAME_LOWER_CASE
+      - ORDER
+      - PROTO3_FIELDS_AVOID_REQUIRED
+      - PROTO3_GROUPS_AVOID
+      - REPEATED_FIELD_NAMES_PLURALIZED
+      - QUOTE_CONSISTENT
+
+  # Linter rules option.
+  rules_option:
+    # MAX_LINE_LENGTH rule option.
+    max_line_length:
+      # Enforces a maximum line length
+      max_chars: 120
+      # Specifies the character count for tab characters
+      tab_chars: 2
+
+    # FILE_NAMES_LOWER_SNAKE_CASE rule option.
+    file_names_lower_snake_case:
+      excludes:
+        - ../proto/invalidFileName.proto
+
+    # QUOTE_CONSISTENT rule option.
+    quote_consistent:
+      # Available quote are "double" or "single".
+      quote: double

.vscode/settings.json

@@ -0,0 +1,20 @@
+{
+    "editor.rulers": [
+        120
+    ],
+    "editor.formatOnSave": true,
+    "[python]": {
+        "editor.defaultFormatter": "ms-python.black-formatter",
+        "editor.formatOnSave": true
+    },
+    "black-formatter.args": [
+        "--line-length",
+        "120"
+    ],
+    "flake8.args": [
+        "--config=setup.cfg"
+    ],
+    "python.testing.unittestEnabled": false,
+    "python.testing.pytestEnabled": false,
+    "python.envFile": "${workspaceFolder}/.env"
+}

CODEOWNERS

@@ -0,0 +1,2 @@
+# Default code owners for the entire repository
+* @w4nderlust

CODE_OF_CONDUCT.md

@@ -0,0 +1,3 @@
+# Code of conduct
+
+Ludwig adopts the [Linux Foundation code of conduct](https://lfprojects.org/policies/code-of-conduct/).

CONTRIBUTING.md

@@ -0,0 +1,220 @@
+# Contributing
+
+Everyone is welcome to contribute, and we value everybody’s contribution. Code is thus not the only
+way to help the community. Answering questions, helping others, reaching out and improving the
+documentation are immensely valuable contributions as well.
+
+It also helps us if you spread the word: reference the library from blog posts on the awesome
+projects it made possible, shout out on X every time it has helped you, or simply star the
+repo to say "thank you".
+
+Check out the official [ludwig docs](https://ludwig-ai.github.io/ludwig-docs/) to get oriented
+around the codebase, and join the community!
+
+## Open Issues
+
+Issues are listed at: <https://github.com/ludwig-ai/ludwig/issues>
+
+If you would like to work on any of them, make sure it is not already assigned to someone else.
+
+You can self-assign it by commenting on the Issue page with one of the keywords: `#take` or
+`#self-assign`.
+
+Work on your self-assigned issue and eventually create a Pull Request.
+
+## Creating Pull Requests
+
+1. Fork the [repository](https://github.com/ludwig-ai/ludwig) by clicking on the "Fork" button on
+   the repository's page. This creates a copy of the code under your GitHub user account.
+
+1. Clone your fork to your local disk, and add the base repository as a remote:
+
+   ```bash
+   git clone git@github.com:<your Github handle>/ludwig.git
+   cd ludwig
+   git remote add upstream https://github.com/ludwig-ai/ludwig.git
+   ```
+
+1. Create a new branch to hold your development changes:
+
+   ```bash
+   git checkout -b a-descriptive-name-for-my-changes
+   ```
+
+   *Do not*\* work on the `master` branch.
+
+1. Set up a development environment by running the following command in a virtual environment:
+
+   ```bash
+   pip install -e .
+   ```
+
+   The above command will install the core dependencies in developer mode. If you would like to
+   be able to potentially make changes to the overall Ludwig codebase, then use the following command:
+
+   ```bash
+   pip install -e .[full]
+   ```
+
+   Please note that in certain Shell environments (e.g., the `Z shell`), the dependencies in brackets have to be quoted:
+
+   ```bash
+   pip install -e ."[full]"
+   ```
+
+   If you do not need access to the entire Ludwig codebase, but just want to be able to run `pytest` on the essential
+   functionality, then you would replace the above command with:
+
+   ```bash
+   pip install -e .[test]
+   ```
+
+   (Please use `pip install -e ."[test]"` where your Shell environment requires quotes around the square brackets.)
+
+   For the full list of the optional dependencies available in Ludwig, please see
+   [Installation Guide](https://ludwig.ai/latest/getting_started/installation/) and `pyproject.toml` in the root of the Ludwig
+   repository.
+
+1. On MacOS with Apple Silicon, if this installation approach runs into errors, you may need to install the following
+   prerequisites:
+
+   ```bash
+   brew install cmake libomp
+   ```
+
+   This step requires `homebrew` to be installed on your development machine.
+
+1. Install and run `pre-commit`:
+
+   ```bash
+   pip install pre-commit
+   pre-commit install
+   ```
+
+1. Develop features on your branch.
+
+1. Format your code by running pre-commits so that your newly added files look nice:
+
+   ```bash
+   pre-commit run
+   ```
+
+   Pre-commits also run automatically when committing.
+
+1. Once you're happy with your changes, make a commit to record your changes locally:
+
+   ```bash
+   git add .
+   git commit
+   ```
+
+   It is a good idea to sync your copy of the code with the original repository regularly. This
+   way you can quickly account for changes:
+
+   ```bash
+   git fetch upstream
+   git rebase upstream/master
+   ```
+
+   Push the changes to your account using:
+
+   ```bash
+   git push -u origin a-descriptive-name-for-my-changes
+   ```
+
+1. Once you are satisfied, go the webpage of your fork on GitHub. Click on "Pull request" to send
+   your contribution to the project maintainers for review.
+
+## Running Tests
+
+Ludwig's test suite lives in `tests/`. The main categories are:
+
+| Directory                  | What it covers                                          |
+| -------------------------- | ------------------------------------------------------- |
+| `tests/ludwig/`            | Unit tests for individual modules (fast, no GPU needed) |
+| `tests/integration_tests/` | End-to-end training/prediction pipelines                |
+| `tests/regression_tests/`  | Accuracy regression checks                              |
+
+**Run the full unit test suite:**
+
+```bash
+pytest tests/ludwig/
+```
+
+**Run a single test file:**
+
+```bash
+pytest tests/ludwig/encoders/test_image_encoder.py -v
+```
+
+**Run integration tests (slow, uses GPUs if available):**
+
+```bash
+pytest tests/integration_tests/ -v
+```
+
+**Run tests matching a keyword:**
+
+```bash
+pytest tests/ -k "test_binary_feature" -v
+```
+
+Tests that require GPUs, Ray, or heavy optional dependencies are marked with pytest marks
+(`@pytest.mark.slow`, `@pytest.mark.distributed`, etc.) and are automatically skipped in CI
+unless the appropriate resources are available.
+
+## Codebase Overview
+
+A quick map of the most important modules:
+
+| Path                           | Purpose                                                                     |
+| ------------------------------ | --------------------------------------------------------------------------- |
+| `ludwig/api.py`                | `LudwigModel` — the main public API (train, predict, evaluate)              |
+| `ludwig/schema/`               | Pydantic v2 config schema — `ModelConfig`, feature configs, trainer configs |
+| `ludwig/features/`             | Feature types: preprocessing, encoding, decoding, metrics                   |
+| `ludwig/encoders/`             | Encoder implementations (text, image, audio, tabular, …)                    |
+| `ludwig/decoders/`             | Decoder implementations                                                     |
+| `ludwig/models/`               | `ECD` (encoder–combiner–decoder) and `LLM` model classes                    |
+| `ludwig/trainers/`             | Trainer implementations (ECD, LLM fine-tune, inference-only)                |
+| `ludwig/data/preprocessing.py` | Tabular data loading and preprocessing pipeline                             |
+| `ludwig/backend/`              | Execution backends (`LocalBackend`, `RayBackend`)                           |
+| `ludwig/utils/data_utils.py`   | File format readers (CSV, Parquet, JSON, …)                                 |
+| `ludwig/collect.py`            | CLI tools: collect activations, weights                                     |
+
+**Key abstractions:**
+
+- `BaseFeature` → `InputFeature` / `OutputFeature` — every feature type inherits from one of these.
+- `Encoder` / `Decoder` — registered via `@register_encoder` / `@register_decoder`; looked up by
+  `encoder.type` / `decoder.type` from config.
+- `DataFormatPreprocessor` — dispatches file loading per format via a strategy (reader function).
+  `FileBasedPreprocessor(read_fn)` covers all tabular file types; `HDF5Preprocessor` and the
+  in-memory `DictPreprocessor` / `DataFramePreprocessor` are separate.
+- `BackendCapabilities` — frozen dataclass of feature flags advertised by each backend.
+- `InferenceOnlyTrainer` (config type `"none"`) — runs evaluation without training; used for
+  zero-shot / few-shot LLM inference.
+
+**Adding a new feature type:**
+
+1. Create `ludwig/features/<name>_feature.py` with `<Name>FeatureMixin`, `<Name>InputFeature`,
+   and/or `<Name>OutputFeature`.
+1. Create `ludwig/schema/features/<name>_feature.py` with the Pydantic config classes.
+1. Register them with `@register_input_feature("<name>")` / `@register_output_feature("<name>")`.
+1. Add a preprocessing config class to `ludwig/schema/features/preprocessing/`.
+1. Add tests in `tests/ludwig/features/test_<name>_feature.py`.
+
+## Other Tips
+
+- Add unit tests for any new code you write.
+- Make sure tests pass. See the [Developer Guide](https://ludwig-ai.github.io/ludwig-docs/latest/developer_guide/style_guidelines_and_tests/) for more details.
+- Keep `except Exception:` blocks narrow: use `except ImportError:` for optional imports and
+  `except pydantic.ValidationError:` for schema fallbacks. Broad exception swallowing makes
+  production bugs invisible.
+
+## Attribution
+
+This contributing guideline is adapted from `huggingface`, available at <https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md>.
+
+## Code of Conduct
+
+Please be mindful of and adhere to the Linux Foundation's
+[Code of Conduct](https://lfprojects.org/policies/code-of-conduct) when contributing to Ludwig.

IMPROVEMENT_PLAN.md

@@ -0,0 +1,302 @@
+# Ludwig Codebase Improvement Plan
+
+Generated: 2026-05-16. Based on a thorough review of the full codebase (110k lines, ~400 files).
+
+---
+
+## Executive Summary
+
+Ludwig is **architecturally sound at the macro level** — the Backend abstraction, modular encoder/decoder registries, and schema-driven config system are genuinely well-designed. The problem is at the **meso level**: a handful of files have become god objects that grow without bound (`preprocessing.py` 2407 lines, `api.py` 2237 lines, `trainer.py` 1766 lines), there are 208 untyped `Any` fields hiding data contracts, 112 bare `except Exception:` handlers silencing failures, and 172 TODOs indicating unresolved design decisions. The single most impactful structural fix is decomposing the 18-class `DataFormatPreprocessor` hierarchy into a reader-strategy pattern (~900 lines of duplication removed). The single most impactful production fix is auditing the 112 bare `except Exception:` handlers.
+
+---
+
+## Critical Issues (Must Fix)
+
+### C1 — `FatherPreprocessor` typo (`preprocessing.py:689`)
+The Feather format preprocessor is named `FatherPregressor`. It is mapped to `FEATHER_FORMATS` at line 1149. Any contributor looking for the Feather reader would never find it.
+- **Fix:** Rename to `FeatherPreprocessor` (class + format map).
+
+### C2 — Duplicate `keys()` on `TrainingStats` (`api.py:150,168`)
+`TrainingStats.keys()` is defined twice. The second definition silently overrides the first; suppressed with `# noqa: F811`. Breaks the Mapping protocol contract.
+- **Fix:** Delete lines 150-153 (the first definition).
+
+### C3 — 18 nearly-identical `DataFormatPreprocessor` subclasses (`preprocessing.py:186-1046`)
+Every subclass (`DictPreprocessor`, `CSVPreprocessor`, `JSONPreprocessor`, `ParquetPreprocessor`, ...) implements `preprocess_for_training()`, `preprocess_for_prediction()`, `prepare_processed_data()` with bodies that differ only in the pandas read call. ~900 lines of structural duplication.
+- **Fix:** See PR-3.
+
+### C4 — Silent failure on missing files (`backend/base.py:191-212`)
+`LocalPreprocessingMixin.read_binary_files()` passes `None` paths to `map_fn` without error. Datasets silently drop samples at scale.
+- **Fix:** Add `None` guard before calling `map_fn`; raise `ValueError` with path context.
+
+### C5 — 112 bare `except Exception:` handlers throughout codebase
+Examples: `except Exception: return None` in `strings_utils.py:138`, `except Exception: print("FAILURE")` in `check.py:32`. Makes production debugging impossible.
+- **Fix:** See PR-7.
+
+### C6 — OOM footgun in `collect_activations()` (`collect.py:186`)
+Marked `# TODO -> Fix OOM on large models e.g. llama 3 8B`. Loads all activations for all samples into RAM at once. Will OOM on any LLM with >10k samples.
+- **Fix:** Stream activations in chunks; write intermediate results to disk.
+
+---
+
+## Major Issues (Should Fix)
+
+### M1 — `dict[str, Any]` type aliases hide all data contracts (`types.py:5-47`)
+All public typedefs (`FeatureConfigDict`, `ModelConfigDict`, `TrainingSetMetadataDict`, etc.) are `dict[str, Any]`. 208 total `Any` usages. Disables IDE completion and static analysis.
+- **Fix:** See PR-1.
+
+### M2 — Trainer is a 1766-line god object (`trainer.py:104-1766`)
+`Trainer` inherits from 4 mixins + `BaseTrainer`. `__init__` has 24 parameters. `train_loop()` is ~240 lines mixing batching, backprop, gradient accumulation, metrics, checkpointing, early stopping.
+- **Fix:** See PR-5, PR-6.
+
+### M3 — `LudwigModel` is a 2237-line god object (`api.py:160-2237`)
+Mixes model lifecycle, serialization, experiment logging, hyperopt, and serving.
+- **Fix:** See PR-8.
+
+### M4 — Feature preprocessing modules have no shared base
+`_ImagePreprocessing`, `_AudioPreprocessing`, `_CategoryPreprocessing` etc. each re-implement missing value handling, dtype casting, reshaping with no common base class.
+- **Fix:** See PR-4.
+
+### M5 — `create_passthrough_input_feature()` is an invisible feature type (`base_feature.py:648-703`)
+Defines an inline class that bypasses the feature registry entirely. Undiscoverable.
+- **Fix:** See PR-9.
+
+### M6 — 92 suppressed type errors (`# type: ignore`, `# noqa`)
+Including bugs: `api.py:288` has `# type: ignore [assignment]` because `self.config_fp = None` when type expects `str` — that's an uninitialised-state bug.
+- **Fix:** Audit all 92; fix root causes.
+
+### M7 — Test coverage gaps on critical paths
+- `preprocessing.py` — no unit tests for individual `DataFormatPreprocessor` subclasses
+- `backend/ray.py:816` — `BatchInferModel` inner class untested
+- `trainers/trainer_dpo.py` — `KTOTrainer`, `ORPOTrainer`, `GRPOTrainer` have no unit tests
+- `collect.py` — zero unit tests
+
+### M8 — Stale TODOs that are actually bugs
+- `trainer.py:302` — "loading an existing model loses metric values" — known data loss on resume
+- `models/base.py:147` — "Remove dummy implementation" — dummy property returns wrong values
+- `api.py:1980` — model type check duplicated between LLM and ECD paths
+
+---
+
+## Minor Issues
+
+### Naming (violating "naming things" rules)
+
+| File:Line | Problem | Fix |
+|-----------|---------|-----|
+| `preprocessing.py:689` | `FatherPreprocessor` — typo for Feather | `FeatherPreprocessor` |
+| `backend/base.py:181` | `LocalPreprocessingMixin` — also handles binary reads | `LocalDataProcessingMixin` |
+| `features/base_feature.py:57` | `BaseFeatureMixin` — vague, has state | `FeaturePreprocessingMixin` |
+| `models/base.py:125` | `ModuleWrapper` — says nothing about purpose | `NonPropertyModuleWrapper` |
+| `trainers/trainer_llm.py:44` | `NoneTrainer` — confusing name for inference-only | `InferenceOnlyTrainer` |
+| `data/cache/manager.py:101` | `CacheManager` — generic Manager anti-pattern | `PreprocessedDataCache` |
+
+### Type Hints
+- `backend/base.py:79` — `capabilities: dict[str, Any]` → `dict[str, bool]`
+- `api.py:145-147` — `TrainingStats` fields → `dict[str, float]`
+- `features/base_feature.py` — 12 methods missing return type annotations
+- `encoders/text_encoders.py:184` — abstract method `get_hf_config_param_names` never enforced
+
+### Docstrings
+- `features/base_feature.py:100` — `add_feature_data()` doesn't describe `proc_df` contract
+- `models/ecd.py:145` — `forward()` doesn't explain `targets` in train vs. predict mode
+- `data/preprocessing.py:143` — `DataFormatPreprocessor` has no docstring for the 3-method contract
+
+### Performance
+- `preprocessing.py:207-214` — converts `dataset` to `pd.DataFrame` 3× in same function
+- `utils/data_utils.py:452` — `hash_dict()` runs `pickle.dumps()` + SHA256 on every call; not cached
+- `features/base_feature.py:178` — `create_sample_input()` generates random tensors every call
+
+### Magic Constants
+- `data/lazy_utils.py:29` — `min(16, (os.cpu_count() or 4) + 4)` — why `+4`? why cap 16? Add comment.
+- `features/image_feature.py:98-99` — ImageNet1K mean/std hardcoded instead of from torchvision
+
+### Dead Code
+- `features/base_feature.py:648-703` — `create_passthrough_input_feature()` inline class factory
+- `utils/visualization_utils.py` — 1568 lines of custom plotting duplicating pandas/plotly
+
+---
+
+## Persona Verdicts
+
+### ML Engineer (Production Pipelines)
+**HIGH RISK** — do not deploy without fixing C4, C5, C6. Silent failures in `read_binary_files()` (C4) silently shrink datasets at scale. 112 bare `except Exception:` (C5) mean production tracebacks are useless. The OOM footgun in `collect.py` (C6) hits every practitioner who runs feature analysis on an LLM. The preprocessing monolith (2407 lines, 18 classes) makes debugging data pipelines require holding an enormous mental model. The trainer mixes concerns so tightly that adding distributed debugging hooks requires touching 6 different mixins.
+
+### ML Researcher (Running Experiments)
+**MODERATE** — good for prototyping, risky for long-running experiments. The declarative YAML config and pydantic schema validation are the right idea, well-executed. However: the known metric-loss-on-resume bug (M8, `trainer.py:302`) is a real reproducibility hazard for multi-day training runs. The HuggingFace encoder schema (`schema/encoders/text_encoders.py`: 2714 lines) is so large that understanding available params for a given model requires reading 100+ lines. Hyperopt is tightly coupled to trainer internals, making custom search spaces fragile across versions.
+
+### Open Source Contributor (First PR)
+**UNWELCOMING** — needs a contributor guide and smaller files. Adding a new feature type requires understanding: (1) `BaseFeatureMixin` vs `InputFeature`/`OutputFeature` split, (2) the schema-vs-feature-module duality (every feature has a matching schema class in `ludwig/schema/features/`), (3) the inner preprocessing module pattern, (4) the encoder/decoder registry. None of this is documented in one place. Feature files are enormous (image: 1378 lines, 66 methods; audio: 675 lines). `create_passthrough_input_feature()` is an invisible feature type that bypasses the registry — a contributor following the normal pattern would never know it exists.
+
+### Social Media ML Reader (HN/Reddit/X)
+**MIXED** — impressive scope, cringe-worthy internals. The feature set is genuinely impressive (600+ datasets, 50+ encoders, Ray distributed, LLM fine-tuning, multimodal). But: `preprocessing.py` has 18 classes with copy-pasted method bodies. `api.py` is 2237 lines. `FatherPreprocessor` (a Feather reader named "Father") has been in production. The `dict[str, Any]` typedefs look like hastily-migrated Python 2 code. 172 TODOs suggest active development paralysis. The architecture deserves better than the execution.
+
+---
+
+## Improvement Plan (Ordered PRs)
+
+### Phase 0 — Quick Wins (1-2 days, zero risk)
+
+**PR-0a: Fix typos + silent bugs** (S)
+- `FatherPreprocessor` → `FeatherPreprocessor` (`preprocessing.py:689,1149`)
+- Remove duplicate `keys()` from `TrainingStats` (`api.py:150-153`)
+- Add `None` guard in `read_binary_files()` (`backend/base.py:207`)
+
+**PR-0b: Fix TODO bugs** (S)
+- Fix metric loss on resume (`trainer.py:302`) — reload metrics from checkpoint
+- Remove fake `input_shape` dummy property (`models/base.py:147`)
+- Fix `check.py:32` silent failure — add `logger.exception()`
+
+---
+
+### Phase 1 — Type System (1 week)
+
+**PR-1: TypedDict for data contracts** (L)
+Replace `dict[str, Any]` aliases in `types.py` with `TypedDict` subclasses. Update callsites. Run mypy; fix revealed type errors.
+- `FeatureConfigDict`, `TrainingSetMetadataDict`, `FeatureMetadataDict` etc.
+- Files: `types.py`, `api.py`, `features/base_feature.py`, `data/preprocessing.py`
+- Impact: ~50 latent bugs caught by mypy; IDE completion for configs
+
+**PR-2: Backend capabilities as frozen dataclass** (S)
+```python
+@dataclass(frozen=True)
+class BackendCapabilities:
+    distributed: bool = False
+    hyperopt: bool = False
+    async_execution: bool = False
+```
+Replace all string-key capability lookups.
+- Files: `backend/base.py`, `backend/ray.py`, `backend/local.py`
+
+---
+
+### Phase 2 — Preprocessing Refactoring (1-2 weeks)
+
+**PR-3: Reader strategy pattern for `DataFormatPreprocessor`** (L)
+Collapse 18 subclasses into one `DataFormatPreprocessor` with injected format reader:
+```python
+class DataFormatReader(ABC):
+    @abstractmethod
+    def read(self, path: str, **kwargs) -> pd.DataFrame: ...
+
+class CSVReader(DataFormatReader):
+    def read(self, path, **kwargs): return pd.read_csv(path, **kwargs)
+
+# One 5-line reader per format instead of one 50-line class per format
+```
+`preprocessing.py` drops from 2407 → ~1000 lines.
+- New package: `ludwig/data/readers/`
+- Add unit tests per reader (normal, missing file, malformed, empty)
+
+**PR-4: Base preprocessing module** (M)
+Extract shared logic (missing value handling, dtype casting, reshaping) into `BasePreprocessingModule`. Have all feature `_Preprocessing` inner classes inherit from it.
+- Files: `features/base_feature.py`, `features/image_feature.py`, `features/audio_feature.py`, `features/category_feature.py`
+
+---
+
+### Phase 3 — Trainer Modularization (1-2 weeks)
+
+**PR-5: Trainer composition over mixin inheritance** (L)
+```python
+# Before: class Trainer(CheckpointMixin, EarlyStoppingMixin, MetricsMixin, ProfilingMixin, BaseTrainer)
+# After:
+class Trainer(BaseTrainer):
+    def __init__(self, config, backend,
+                 checkpointer: CheckpointService,
+                 early_stopper: EarlyStoppingService,
+                 metrics_collector: MetricsCollectionService,
+                 profiler: ProfilingService | None = None): ...
+```
+- `Trainer.__init__` shrinks from 24 params to 5
+- Each service is independently testable
+- Files: `trainers/trainer.py`, `trainers/mixins.py` → `trainers/services/`
+
+**PR-6: Decompose `train_loop()`** (M)
+Break 240-line method into `_forward_pass()`, `_backward_pass()`, `_update_metrics()`, `_maybe_checkpoint()`, `_maybe_early_stop()` — each <50 lines.
+- Files: `trainers/trainer.py`
+
+---
+
+### Phase 4 — Error Handling (1 week)
+
+**PR-7: Fix bare `except Exception:` handlers** (M)
+112 instances. Replace with specific exception types + logging. At minimum:
+- `check.py:32` — add `logger.exception()`
+- `strings_utils.py:138` — replace `return None` with typed exception
+- `image_utils.py:98` — replace `return None` with logged warning
+- All handlers that discard errors silently in preprocessing/data loading paths
+
+**PR-7b: Fix `collect_activations()` OOM** (M)
+Add chunked streaming with disk offload. `collect.py`.
+
+---
+
+### Phase 5 — Structure & Dead Code (1 week)
+
+**PR-8: Split `LudwigModel` (`api.py`)** (XL)
+```
+api.py (~600 lines) — train(), evaluate(), predict(), save(), load() only
+experiment.py    — run_experiment(), hyperopt integration
+serve_v2.py      — already partially extracted; complete it
+explain.py       — already partially extracted
+```
+
+**PR-9: Promote `PassthroughInputFeature`** (S)
+Remove `create_passthrough_input_feature()` factory (`base_feature.py:648-703`). Create `features/passthrough_feature.py` with a proper class in the feature registry.
+
+**PR-10: Rename misleading identifiers** (S)
+`NoneTrainer` → `InferenceOnlyTrainer`, `LocalPreprocessingMixin` → `LocalDataProcessingMixin`, `CacheManager` → `PreprocessedDataCache`
+
+---
+
+### Phase 6 — Test Coverage (Ongoing)
+
+**PR-11: Unit tests for format readers** (M)
+After PR-3: `tests/ludwig/data/readers/test_*.py` — normal read, missing file, malformed, empty.
+
+**PR-12: Unit tests for `collect.py`** (S)
+Test with 2-layer model; verify output shapes; verify chunked mode.
+
+**PR-13: Unit tests for `BatchInferModel`** (S)
+Test inner class at `backend/ray.py:816`.
+
+**PR-14: Unit tests for RLHF trainers** (M)
+`KTOTrainer`, `ORPOTrainer`, `GRPOTrainer` — unit tests with tiny models, no GPU required.
+
+---
+
+### Phase 7 — Documentation & Contributor Experience (1 week)
+
+**PR-15: Feature contributor guide** (S)
+`docs/developer_guide/adding_a_feature_type.md` — schema class, feature module, mixin pattern, required vs optional methods, test template.
+
+**PR-16: Resolve all TODOs** (M)
+Audit all 172. Fix bugs; convert design questions to GitHub issues; delete the comment.
+
+---
+
+## Summary Table
+
+| PR | Description | Size | Priority |
+|----|-------------|------|----------|
+| PR-0a | Fix typos + silent bugs | S | P0 |
+| PR-0b | Fix TODO bugs | S | P0 |
+| PR-7 | Fix bare `except Exception:` | M | P0 |
+| PR-7b | Fix `collect_activations()` OOM | M | P0 |
+| PR-1 | TypedDict for data contracts | L | P0 |
+| PR-3 | Reader strategy for format preprocessors | L | P0 |
+| PR-2 | Backend capabilities dataclass | S | P1 |
+| PR-4 | Base preprocessing module | M | P1 |
+| PR-5 | Trainer composition over inheritance | L | P1 |
+| PR-6 | Decompose `train_loop()` | M | P1 |
+| PR-11 | Unit tests for format readers | M | P1 |
+| PR-12 | Unit tests for `collect.py` | S | P1 |
+| PR-13 | Unit tests for `BatchInferModel` | S | P1 |
+| PR-14 | Unit tests for RLHF trainers | M | P1 |
+| PR-8 | Split `LudwigModel` | XL | P2 |
+| PR-9 | Promote `PassthroughInputFeature` | S | P2 |
+| PR-10 | Rename misleading identifiers | S | P2 |
+| PR-15 | Feature contributor guide | S | P2 |
+| PR-16 | Resolve all TODOs | M | P2 |
+
+**Recommended start order:** PR-0a → PR-7 → PR-3 → PR-1 → PR-5
+**Total estimated effort:** 6-8 weeks of focused engineering (PRs within phases can be parallelized).

LICENSE

@@ -0,0 +1,252 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+
+--------------------------------------------------------------------------
+
+Code in ludwig/api_annotations.py adapted from
+https://github.com/ray-project/ray (Apache-2.0 License)
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    https://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+--------------------------------------------------------------------------
+
+Code in ludwig/utils/structural_warnings.py adapted from
+https://github.com/ray-project/ray (Apache-2.0 License)
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    https://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+--------------------------------------------------------------------------
+
+Code in ludwig/utils/logging_utils.py adapted from
+https://github.com/ray-project/ray (Apache-2.0 License)
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    https://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

NOTICE

@@ -0,0 +1,107 @@
+Ludwig includes derived work from TensorFlow(https://github.com/tensorflow/tensorflow) under the Apache License 2.0:
+
+Copyright 2016 The prometheus-operator Authors
+
+
+Licensed under the Apache License, Version 2.0 (the "License");
+
+you may not use this file except in compliance with the License.
+
+You may obtain a copy of the License at
+
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+
+Unless required by applicable law or agreed to in writing, software
+
+distributed under the License is distributed on an "AS IS" BASIS,
+
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+
+See the License for the specific language governing permissions and
+
+limitations under the License.
+
+
+The derived work can be found in the files: ludwig/models/modules/convolutional_modules.py
+
+
+------
+
+Ludwig includes derived work from Keras(https://github.com/keras-team/keras) under the MIT License:
+
+
+COPYRIGHT
+
+All contributions by François Chollet:
+
+Copyright (c) 2015 - 2018, François Chollet.
+
+All rights reserved.
+
+
+All contributions by Google:
+
+Copyright (c) 2015 - 2018, Google, Inc.
+
+All rights reserved.
+
+
+All contributions by Microsoft:
+
+Copyright (c) 2017 - 2018, Microsoft, Inc.
+
+All rights reserved.
+
+
+All other contributions:
+
+Copyright (c) 2015 - 2018, the respective contributors.
+
+All rights reserved.
+
+
+Each contributor holds copyright over their respective contributions.
+
+The project versioning (Git) records all such contribution source information.
+
+LICENSE
+
+The MIT License (MIT)
+
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+
+of this software and associated documentation files (the "Software"), to deal
+
+in the Software without restriction, including without limitation the rights
+
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+
+copies of the Software, and to permit persons to whom the Software is
+
+furnished to do so, subject to the following conditions:
+
+
+The above copyright notice and this permission notice shall be included in all
+
+copies or substantial portions of the Software.
+
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+
+SOFTWARE.
+
+
+The derived work can be found in the files: mkdocs/code_docs_autogen.py

README.md

@@ -0,0 +1,353 @@
+<p align="center">
+  <a href="https://ludwig.ai">
+    <img src="https://github.com/ludwig-ai/ludwig-docs/raw/main/docs/images/ludwig_hero_smaller.jpg" height="150">
+  </a>
+</p>
+
+<div align="center">
+
+**Declarative deep learning framework for LLMs, multimodal models, and tabular AI.**
+
+[![PyPI version](https://badge.fury.io/py/ludwig.svg)](https://badge.fury.io/py/ludwig)
+[![Discord](https://img.shields.io/badge/Discord-Join%20Chat-5865F2?logo=discord&logoColor=white)](https://discord.gg/CBgdrGnZjy)
+[![DockerHub](https://img.shields.io/docker/pulls/ludwigai/ludwig.svg)](https://hub.docker.com/r/ludwigai)
+[![Downloads](https://pepy.tech/badge/ludwig)](https://pepy.tech/project/ludwig)
+[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/ludwig-ai/ludwig/blob/main/LICENSE)
+[![X](https://img.shields.io/twitter/follow/ludwig_ai.svg?style=social&logo=twitter)](https://twitter.com/ludwig_ai)
+
+[**Docs**](https://ludwig.ai) · [**Getting Started**](https://ludwig.ai/latest/getting_started/) · [**Examples**](https://ludwig.ai/latest/examples) · [**Discord**](https://discord.gg/CBgdrGnZjy)
+
+</div>
+
+______________________________________________________________________
+
+## What is Ludwig?
+
+Ludwig is a **declarative deep learning framework** that lets you train, fine-tune, and deploy AI models — from LLM fine-tuning to tabular classification — using a YAML config file and zero boilerplate Python.
+
+```yaml
+# Fine-tune Llama-3.1 with LoRA in one config file
+model_type: llm
+base_model: meta-llama/Llama-3.1-8B
+adapter:
+  type: lora
+trainer:
+  type: finetune
+  epochs: 3
+input_features:
+  - name: instruction
+    type: text
+output_features:
+  - name: response
+    type: text
+```
+
+```bash
+ludwig train --config model.yaml --dataset my_data.csv
+```
+
+**Tech stack:** Python 3.12 · PyTorch 2.7+ · Pydantic 2 · Transformers 5 · Ray 2.54
+
+Ludwig is hosted by the [Linux Foundation AI & Data](https://lfaidata.foundation/).
+
+______________________________________________________________________
+
+## What's New in Ludwig 0.16
+
+| Feature                         | Description                                                                                            |
+| ------------------------------- | ------------------------------------------------------------------------------------------------------ |
+| **PatchTST & N-BEATS encoders** | State-of-the-art timeseries forecasting encoders with MASE/sMAPE metrics                               |
+| **Advanced PEFT adapters**      | PiSSA, EVA, CorDA/LoftQ initializers; TinyLoRA, OFT, HRA, WaveFT, LN-Tuning, VBLoRA, C3A adapter types |
+| **VLM fine-tuning**             | Train LLaVA, Qwen2-VL, InternVL via `is_multimodal: true` with gated cross-attention                   |
+| **HyperNetwork combiner**       | Conditioning-based feature fusion — one feature generates weights for others                           |
+| **Nash-MTL & Pareto-MTL**       | Game-theoretic and preference-based multi-task loss balancing                                          |
+| **LLM config generation**       | `ludwig generate_config "describe your task"` — LLM writes the YAML for you                            |
+| **ModelInspector**              | Architecture analysis, weight collection, feature importance proxy                                     |
+| **Ray Serve & KServe**          | Distributed and Kubernetes-native model deployment shims                                               |
+| **GRPO alignment**              | Reward-model-free RLHF via Group Relative Policy Optimization                                          |
+| **torchao quantization + QAT**  | PyTorch-native `int4/int8/float8` with Quantization-Aware Training                                     |
+| **Multi-adapter PEFT**          | Multiple named LoRA adapters with weighted merging (TIES, DARE, SVD)                                   |
+| **Native Optuna executor**      | GPT/TPE/CMA-ES samplers, pruning, resumable SQLite/PostgreSQL storage                                  |
+| **Timeseries forecasting**      | `model.forecast(dataset, horizon=N)` API with `TimeseriesOutputFeature`                                |
+| **Muon & ScheduleFreeAdamW**    | New optimizers for large-scale pretraining and fine-tuning                                             |
+| **Image segmentation decoders** | UNet, SegFormer, FPN decoders for semantic segmentation                                                |
+
+______________________________________________________________________
+
+## Installation
+
+```bash
+pip install ludwig           # core
+pip install ludwig[full]     # all optional dependencies
+pip install ludwig[llm]      # LLM fine-tuning only
+```
+
+Requires Python 3.12+. See [contributing](https://github.com/ludwig-ai/ludwig/blob/main/CONTRIBUTING.md) for a full dependency matrix.
+
+______________________________________________________________________
+
+## Quick Start
+
+### Fine-tune an LLM (instruction tuning)
+
+[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1c3AO8l_H6V_x37RwQ8V7M6A-RmcBf2tG?usp=sharing)
+
+Ludwig supports the full LLM fine-tuning spectrum:
+
+| Technique                         | Config key                                                               |
+| --------------------------------- | ------------------------------------------------------------------------ |
+| Supervised fine-tuning (SFT)      | `trainer.type: finetune`                                                 |
+| DPO / KTO / ORPO / GRPO alignment | `trainer.type: dpo` (or `kto`, `orpo`, `grpo`)                           |
+| LoRA / DoRA / VeRA / PiSSA        | `adapter.type: lora` (or `dora`, `vera`, `lora` + `init_weights: pissa`) |
+| 4-bit QLoRA (bitsandbytes)        | `quantization.bits: 4`                                                   |
+| torchao + QAT                     | `quantization.backend: torchao`                                          |
+| Multi-adapter with merging        | `adapters:` dict + `merge:` block                                        |
+| VLM (vision-language)             | `is_multimodal: true`                                                    |
+
+```yaml
+model_type: llm
+base_model: meta-llama/Llama-3.1-8B
+
+quantization:
+  bits: 4
+
+adapter:
+  type: lora
+
+prompt:
+  template: |
+    ### Instruction: {instruction}
+    ### Input: {input}
+    ### Response:
+
+input_features:
+  - name: prompt
+    type: text
+
+output_features:
+  - name: output
+    type: text
+
+trainer:
+  type: finetune
+  learning_rate: 0.0001
+  batch_size: 1
+  gradient_accumulation_steps: 16
+  epochs: 3
+  learning_rate_scheduler:
+    decay: cosine
+    warmup_fraction: 0.01
+
+backend:
+  type: local
+```
+
+```bash
+export HUGGING_FACE_HUB_TOKEN="<your_token>"
+ludwig train --config model.yaml --dataset "ludwig://alpaca"
+```
+
+### Train a multimodal classifier
+
+```yaml
+input_features:
+  - name: review_text
+    type: text
+    encoder:
+      type: bert
+  - name: star_rating
+    type: number
+  - name: product_image
+    type: image
+    encoder:
+      type: dinov2
+
+output_features:
+  - name: recommended
+    type: binary
+```
+
+```bash
+ludwig train --config model.yaml --dataset reviews.csv
+```
+
+### Generate a config from natural language
+
+```bash
+ludwig generate_config "I have a CSV with age, income, education level, and I want to predict loan default"
+```
+
+### Make predictions
+
+```bash
+ludwig predict --model_path results/experiment_run/model --dataset new_data.csv
+```
+
+### Launch a REST API
+
+```bash
+ludwig serve --model_path results/experiment_run/model
+# POST http://localhost:8000/predict
+```
+
+______________________________________________________________________
+
+## Capabilities
+
+<details>
+<summary><strong>LLM Fine-Tuning</strong></summary>
+
+- **Supervised fine-tuning (SFT)** on instruction/response pairs
+- **Alignment training**: DPO, KTO, ORPO, GRPO (reward-model-free RLHF)
+- **PEFT adapters**: LoRA, DoRA, VeRA, LoRA+, TinyLoRA, OFT, HRA, WaveFT, LN-Tuning, VBLoRA, C3A
+- **LoRA initializers**: PiSSA, EVA, CorDA, LoftQ for improved convergence
+- **Multi-adapter PEFT**: multiple named adapters on one base model, switchable at runtime; merge with TIES, DARE, SVD, magnitude pruning
+- **Quantization**: 4-bit/8-bit QLoRA (bitsandbytes), torchao int4/int8/float8 with QAT
+- **VLM fine-tuning**: LLaVA, Qwen2-VL, InternVL via `is_multimodal: true`
+- **Sequence packing** for efficient training on variable-length inputs
+- **Paged and 8-bit optimizers** for memory-efficient training
+
+</details>
+
+<details>
+<summary><strong>Multimodal & Tabular Models</strong></summary>
+
+- **Input modalities**: text, numbers, categories, binary, sets, bags, sequences, images, audio, timeseries, vectors, dates
+- **Text encoders**: any HuggingFace Transformer (BERT, RoBERTa, ModernBERT, Qwen3, Llama-3.1, etc.), plus Mamba-2, Jamba
+- **Image encoders**: DINOv2, ConvNeXt, EfficientNet, ViT, CAFormer, ConvFormer, PoolFormer, TIMM (1000+ models)
+- **Timeseries encoders**: PatchTST, N-BEATS, CNN, RNN, Transformer; MASE and sMAPE metrics; `model.forecast()` API
+- **Combiners**: concat, transformer, tab_transformer, FT-Transformer, TabNet, TabPFN v2, HyperNetwork, ProjectAggregate, GatedFusion, Perceiver
+- **Multi-task learning**: multiple output features in a single model; Nash-MTL, Pareto-MTL, FAMO, GradNorm, uncertainty loss balancing
+- **Image segmentation**: UNet, SegFormer, FPN decoders
+
+</details>
+
+<details>
+<summary><strong>Training Infrastructure</strong></summary>
+
+- **Distributed training**: HuggingFace Accelerate with DDP, FSDP, DeepSpeed (zero-code changes)
+- **Ray backend**: training across a Ray cluster, larger-than-memory datasets via Ray Data
+- **Automatic batch size selection** and learning rate range test
+- **Mixed precision** (fp16/bf16), gradient checkpointing, gradient accumulation
+- **Optimizers**: AdamW, Adafactor, SGD, Muon, ScheduleFreeAdamW, Lion, paged/8-bit variants
+- **Learning rate schedulers**: cosine, linear, polynomial, reduce-on-plateau, OneCycleLR
+- **Model Soup**: uniform and greedy checkpoint averaging for better generalization at zero inference cost
+- **Modality dropout** for robust multimodal models
+
+</details>
+
+<details>
+<summary><strong>Hyperparameter Optimization</strong></summary>
+
+- **Executors**: Ray Tune (ASHA, PBT, Bayesian) and native Optuna (auto/GP/TPE/CMA-ES)
+- **Optuna persistence**: SQLite or PostgreSQL for resumable HPO runs
+- **Pruning** with Optuna's MedianPruner and HyperbandPruner
+- **Search spaces**: uniform, log-uniform, choice, randint, quantized
+- **Full Ludwig config** is searchable — any nested parameter can be a hyperparameter
+
+</details>
+
+<details>
+<summary><strong>Production & Deployment</strong></summary>
+
+- **REST API**: FastAPI server with Prometheus metrics and structured logging (`ludwig serve`)
+- **vLLM serving**: OpenAI-compatible API with PagedAttention and continuous batching
+- **Ray Serve**: distributed deployment with auto-scaling and traffic splitting
+- **KServe**: Kubernetes-native deployment with Open Inference Protocol v2
+- **Model export**: SafeTensors (default), `torch.export` `.pt2` bundles, ONNX
+- **HuggingFace Hub**: `ludwig upload hf_hub` — push model + auto-generated model card
+- **Docker**: prebuilt containers at [ludwigai/ludwig](https://hub.docker.com/u/ludwigai)
+
+</details>
+
+<details>
+<summary><strong>Tooling & Integrations</strong></summary>
+
+- **Experiment tracking**: TensorBoard, Weights & Biases, Comet ML, MLflow, Aim Stack
+- **Model inspection**: `ModelInspector` — weight enumeration, architecture summary, feature importance proxy
+- **Visualizations**: learning curves, confusion matrices, calibration plots, ROC curves, hyperopt analysis
+- **AutoML**: `ludwig.automl.auto_train()` — give it a dataset and a time budget; the YAML-driven search space samples encoder/combiner/decoder combinations and validates them before training
+- **Dataset quality checks**: `from ludwig.utils.dataset_quality import check_dataset_quality` — validates a DataFrame before training (missing values, class imbalance, near-duplicate columns, ID leakage, …)
+- **OpenML integration**: load any OpenML task directly — `OpenMLLoader` fetches by task ID and caches locally as Parquet
+- **LLM config generation**: `ludwig generate_config "describe your task"` — LLM writes the YAML
+- **K-fold cross-validation**: `ludwig experiment --k_fold N`
+- **Dataset Zoo**: 70+ built-in benchmark datasets (`ludwig://mnist`, `ludwig://alpaca`, …)
+
+</details>
+
+______________________________________________________________________
+
+## Examples
+
+### LLM & Alignment
+
+| Use Case                              | Link                                                                                |
+| ------------------------------------- | ----------------------------------------------------------------------------------- |
+| LLM instruction tuning (LoRA + QLoRA) | [examples/llm](https://ludwig.ai/latest/examples/llm/llm_finetuning)                |
+| DPO / GRPO alignment                  | [examples/llm/alignment](https://ludwig.ai/latest/examples/llm/alignment)           |
+| Advanced PEFT (PiSSA, OFT, VBLoRA, …) | [examples/llms/peft_advanced](https://ludwig.ai/latest/examples/llms/peft_advanced) |
+| VLM fine-tuning (LLaVA, Qwen2-VL)     | [examples/vlm](https://github.com/ludwig-ai/ludwig/tree/main/examples/vlm)          |
+
+### Tabular & Multimodal
+
+| Use Case                               | Link                                                                                              |
+| -------------------------------------- | ------------------------------------------------------------------------------------------------- |
+| Binary classification (Titanic)        | [examples/titanic](https://ludwig.ai/latest/examples/titanic)                                     |
+| Tabular classification (census income) | [examples/adult_census_income](https://ludwig.ai/latest/examples/adult_census_income)             |
+| Multimodal classification              | [examples/multimodal_classification](https://ludwig.ai/latest/examples/multimodal_classification) |
+| Multi-task learning                    | [examples/multi_task](https://ludwig.ai/latest/examples/multi_task)                               |
+
+### Timeseries & Vision
+
+| Use Case                                   | Link                                                                                      |
+| ------------------------------------------ | ----------------------------------------------------------------------------------------- |
+| Timeseries forecasting (PatchTST, N-BEATS) | [examples/forecasting](https://ludwig.ai/latest/examples/forecasting)                     |
+| Weather forecasting                        | [examples/weather](https://ludwig.ai/latest/examples/weather)                             |
+| Image classification (MNIST)               | [examples/mnist](https://ludwig.ai/latest/examples/mnist)                                 |
+| Semantic segmentation                      | [examples/semantic_segmentation](https://ludwig.ai/latest/examples/semantic_segmentation) |
+
+### NLP & Audio
+
+| Use Case                 | Link                                                                                    |
+| ------------------------ | --------------------------------------------------------------------------------------- |
+| Text classification      | [examples/text_classification](https://ludwig.ai/latest/examples/text_classification)   |
+| Named entity recognition | [examples/ner_tagging](https://ludwig.ai/latest/examples/ner_tagging)                   |
+| Machine translation      | [examples/machine_translation](https://ludwig.ai/latest/examples/machine_translation)   |
+| Speech recognition       | [examples/speech_recognition](https://ludwig.ai/latest/examples/speech_recognition)     |
+| Speaker verification     | [examples/speaker_verification](https://ludwig.ai/latest/examples/speaker_verification) |
+
+______________________________________________________________________
+
+## Why Ludwig?
+
+- **Zero boilerplate** — no training loop, no data pipeline, no evaluation code. The YAML config is the entire program.
+- **Best-in-class LLM support** — full spectrum from LoRA to GRPO alignment, torchao QAT, and VLM fine-tuning, all in config.
+- **Multimodal out of the box** — mix text, images, numbers, audio, and timeseries with one config change.
+- **Scale without code changes** — go from laptop → multi-GPU → Ray cluster by changing `backend.type`.
+- **Expert control when you need it** — every activation function, scheduler, and optimizer is configurable.
+- **Reproducible research** — every run is logged and the full config is saved. Compare experiments with `ludwig visualize`.
+
+______________________________________________________________________
+
+## Publications
+
+- [Ludwig: A Type-Based Declarative Deep Learning Toolbox](https://arxiv.org/pdf/1909.07930.pdf) (2019)
+- [Declarative Machine Learning Systems](https://arxiv.org/pdf/2107.08148.pdf) (2021)
+- [Ludwig's State-of-the-Art Benchmarks](https://openreview.net/pdf?id=hwjnu6qW7E4)
+
+______________________________________________________________________
+
+## Community
+
+[![Discord](https://img.shields.io/badge/Discord-Join%20Chat-5865F2?logo=discord&logoColor=white)](https://discord.gg/CBgdrGnZjy)
+
+- [Discord](https://discord.gg/CBgdrGnZjy) — ask questions, share what you've built
+- [GitHub Issues](https://github.com/ludwig-ai/ludwig/issues) — bugs and feature requests
+- [X / Twitter](https://twitter.com/ludwig_ai) — announcements
+- [Medium](https://medium.com/ludwig-ai) — tutorials and deep-dives
+
+<a href="https://github.com/ludwig-ai/ludwig/graphs/contributors">
+  <img src="https://contrib.rocks/image?repo=ludwig-ai/ludwig" />
+</a>
+
+[![Star History Chart](https://api.star-history.com/svg?repos=ludwig-ai/ludwig&type=Date)](https://star-history.com/#ludwig-ai/ludwig&Date)

README.wehub.md

@@ -0,0 +1,7 @@
+# WeHub 来源说明
+
+- 原始项目：`ludwig-ai/ludwig`
+- 原始仓库：https://github.com/ludwig-ai/ludwig
+- 导入方式：上游默认分支的最新快照
+- 原作者、版权和许可证信息以原始仓库及本仓库 LICENSE 为准
+- 本文件仅用于记录来源，不代表 WeHub 是原项目作者

README_KR.md

@@ -0,0 +1,311 @@
+<p align="center">
+  <a href="https://ludwig.ai">
+    <img src="https://github.com/ludwig-ai/ludwig-docs/raw/main/docs/images/ludwig_hero_smaller.jpg" height="150">
+  </a>
+</p>
+
+<div align="center">
+
+_확장성과 효율성을 위해 설계된 선언적 딥러닝 프레임워크_
+
+[![PyPI version](https://badge.fury.io/py/ludwig.svg)](https://badge.fury.io/py/ludwig)
+[![Discord](https://img.shields.io/badge/Discord-Join%20Chat-5865F2?logo=discord&logoColor=white)](https://discord.gg/CBgdrGnZjy)
+[![DockerHub](https://img.shields.io/docker/pulls/ludwigai/ludwig.svg)](https://hub.docker.com/r/ludwigai)
+[![Downloads](https://pepy.tech/badge/ludwig)](https://pepy.tech/project/ludwig)
+[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/ludwig-ai/ludwig/blob/main/LICENSE)
+[![X](https://img.shields.io/twitter/follow/ludwig_ai.svg?style=social&logo=twitter)](https://twitter.com/ludwig_ai)
+
+</div>
+
+# 📖 Ludwig란?
+
+Ludwig는 **LLM** 및 기타 심층 신경망과 같은 **맞춤형** AI 모델을 구축하기 위한 **로우코드** 프레임워크입니다.
+
+주요 기능:
+
+- 🛠 **손쉬운 맞춤형 모델 구축:** 선언적 YAML 설정 파일만으로 최신 LLM을 데이터에 맞춰 학습시킬 수 있습니다. 멀티태스크 및 멀티모달 학습을 지원합니다. 포괄적인 설정 검증으로 잘못된 매개변수 조합을 감지하고 런타임 오류를 방지합니다.
+- ⚡ **확장성과 효율성 최적화:** 자동 배치 크기 선택, 분산 학습([DDP](https://pytorch.org/tutorials/beginner/ddp_series_theory.html), [DeepSpeed](https://github.com/microsoft/DeepSpeed)), 매개변수 효율적 미세 조정([PEFT](https://github.com/huggingface/peft)), 4비트 양자화(QLoRA), 페이지 및 8비트 옵티마이저, 메모리 초과 데이터셋 지원.
+- 📐 **전문가 수준의 제어:** 활성화 함수까지 모델을 완전히 제어할 수 있습니다. 하이퍼파라미터 최적화, 설명 가능성, 풍부한 메트릭 시각화를 지원합니다.
+- 🧱 **모듈식 및 확장 가능:** 설정에서 몇 가지 매개변수만 변경하여 다양한 모델 아키텍처, 태스크, 피처, 모달리티를 실험할 수 있습니다. 딥러닝을 위한 빌딩 블록이라고 생각하세요.
+- 🚢 **프로덕션을 위한 설계:** 사전 빌드된 [Docker](https://hub.docker.com/u/ludwigai) 컨테이너, [Kubernetes](https://github.com/ray-project/kuberay)에서 [Ray](https://www.ray.io/) 실행 네이티브 지원, [Torchscript](https://pytorch.org/docs/stable/jit.html) 및 [Triton](https://developer.nvidia.com/triton-inference-server)으로 모델 내보내기, 한 번의 명령으로 [HuggingFace](https://huggingface.co/models)에 업로드.
+
+Ludwig는 [Linux Foundation AI & Data](https://lfaidata.foundation/)에서 호스팅합니다.
+
+**기술 스택:** Python 3.12 | PyTorch 2.6 | Pydantic 2 | Transformers 5 | Ray 2.54
+
+![img](https://raw.githubusercontent.com/ludwig-ai/ludwig-docs/master/docs/images/ludwig_legos_unanimated.gif)
+
+# 💾 설치
+
+PyPI에서 설치합니다. Ludwig는 Python 3.12 이상을 요구합니다.
+
+```shell
+pip install ludwig
+```
+
+모든 선택적 의존성을 포함하여 설치:
+
+```shell
+pip install ludwig[full]
+```
+
+더 자세한 설치 방법은 [기여 가이드](https://github.com/ludwig-ai/ludwig/blob/main/CONTRIBUTING.md)를 참조하세요.
+
+# 🚂 시작하기
+
+Ludwig의 기능을 빠르게 살펴보고 싶으시다면 이 Colab 노트북을 확인하세요 🚀 [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1lB4ALmEyvcMycE3Mlnsd7I3bc0zxvk39)
+
+LLM 미세 조정을 원하시나요? 다음 노트북을 확인하세요:
+
+1. Fine-Tune Llama-2-7b: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1r4oSEwRJpYKBPM0M0RSh0pBEYK_gBKbe)
+1. Fine-Tune Llama-2-13b: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1zmSEzqZ7v4twBrXagj1TE_C--RNyVAyu)
+1. Fine-Tune Mistral-7b: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1i_8A1n__b7ljRWHzIsAdhO7u7r49vUm4)
+
+전체 튜토리얼은 공식 [시작 가이드](https://ludwig.ai/latest/getting_started/)를 확인하시거나, 엔드투엔드 [예제](https://ludwig.ai/latest/examples)를 살펴보세요.
+
+## 대규모 언어 모델 미세 조정
+
+[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1c3AO8l_H6V_x37RwQ8V7M6A-RmcBf2tG?usp=sharing)
+
+사전 학습된 LLM을 챗봇처럼 지시를 따르도록 미세 조정("인스트럭션 튜닝")해 봅시다.
+
+### 사전 요구 사항
+
+- [HuggingFace API 토큰](https://huggingface.co/docs/hub/security-tokens)
+- 선택한 베이스 모델에 대한 접근 승인 (예: [Llama-3.1-8B](https://huggingface.co/meta-llama/Llama-3.1-8B))
+- 최소 12 GiB VRAM의 GPU (테스트에서는 Nvidia T4를 사용했습니다)
+
+### 실행
+
+[Stanford Alpaca](https://crfm.stanford.edu/2023/03/13/alpaca.html) 데이터셋을 사용합니다. 다음과 같은 테이블 형식의 파일로 구성됩니다:
+
+|                    instruction                    |      input       |                      output                       |
+| :-----------------------------------------------: | :--------------: | :-----------------------------------------------: |
+|       Give three tips for staying healthy.        |                  | 1.Eat a balanced diet and make sure to include... |
+| Arrange the items given below in the order to ... | cake, me, eating |                  I eating cake.                   |
+| Write an introductory paragraph about a famous... |  Michelle Obama  | Michelle Obama is an inspirational woman who r... |
+|                        ...                        |       ...        |                        ...                        |
+
+`model.yaml`이라는 YAML 설정 파일을 다음 내용으로 생성하세요:
+
+```yaml
+model_type: llm
+base_model: meta-llama/Llama-3.1-8B
+
+quantization:
+  bits: 4
+
+adapter:
+  type: lora
+
+prompt:
+  template: |
+    Below is an instruction that describes a task, paired with an input that may provide further context.
+    Write a response that appropriately completes the request.
+
+    ### Instruction:
+    {instruction}
+
+    ### Input:
+    {input}
+
+    ### Response:
+
+input_features:
+  - name: prompt
+    type: text
+
+output_features:
+  - name: output
+    type: text
+
+trainer:
+  type: finetune
+  learning_rate: 0.0001
+  batch_size: 1
+  gradient_accumulation_steps: 16
+  epochs: 3
+  learning_rate_scheduler:
+    decay: cosine
+    warmup_fraction: 0.01
+
+preprocessing:
+  sample_ratio: 0.1
+
+backend:
+  type: local
+```
+
+이제 모델을 학습시켜 봅시다:
+
+```bash
+export HUGGING_FACE_HUB_TOKEN = "<api_token>"
+
+ludwig train --config model.yaml --dataset "ludwig://alpaca"
+```
+
+## 지도 학습 ML
+
+[Rotten Tomatoes](https://www.kaggle.com/stefanoleone992/rotten-tomatoes-movies-and-critic-reviews-dataset) 영화 평론가의 리뷰가 긍정적인지 부정적인지 예측하는 신경망을 만들어 봅시다.
+
+데이터셋은 다음과 같은 CSV 파일입니다:
+
+|     movie_title      | content_rating |              genres              | runtime | top_critic | review_content                                                                                                                                                                                                   | recommended |
+| :------------------: | :------------: | :------------------------------: | :-----: | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------- |
+| Deliver Us from Evil |       R        |    Action & Adventure, Horror    |  117.0  | TRUE       | Director Scott Derrickson and his co-writer, Paul Harris Boardman, deliver a routine procedural with unremarkable frights.                                                                                       | 0           |
+|       Barbara        |     PG-13      | Art House & International, Drama |  105.0  | FALSE      | Somehow, in this stirring narrative, Barbara manages to keep hold of her principles, and her humanity and courage, and battles to save a dissident teenage girl whose life the Communists are trying to destroy. | 1           |
+|   Horrible Bosses    |       R        |              Comedy              |  98.0   | FALSE      | These bosses cannot justify either murder or lasting comic memories, fatally compromising a farce that could have been great but ends up merely mediocre.                                                        | 0           |
+|         ...          |      ...       |               ...                |   ...   | ...        | ...                                                                                                                                                                                                              | ...         |
+
+[여기](https://ludwig.ai/latest/data/rotten_tomatoes.csv)에서 데이터셋 샘플을 다운로드하세요.
+
+```bash
+wget https://ludwig.ai/latest/data/rotten_tomatoes.csv
+```
+
+다음으로 `model.yaml`이라는 YAML 설정 파일을 생성하세요:
+
+```yaml
+input_features:
+  - name: genres
+    type: set
+    preprocessing:
+      tokenizer: comma
+  - name: content_rating
+    type: category
+  - name: top_critic
+    type: binary
+  - name: runtime
+    type: number
+  - name: review_content
+    type: text
+    encoder:
+      type: embed
+output_features:
+  - name: recommended
+    type: binary
+```
+
+이게 전부입니다! 이제 모델을 학습시켜 봅시다:
+
+```bash
+ludwig train --config model.yaml --dataset rotten_tomatoes.csv
+```
+
+**즐거운 모델링 되세요**
+
+Ludwig를 여러분의 데이터에 적용해 보세요. 질문이 있으시면 [Discord에서 문의](https://discord.gg/CBgdrGnZjy)해 주세요.
+
+# ❓ Ludwig를 사용해야 하는 이유
+
+- **최소한의 머신러닝 보일러플레이트**
+
+  Ludwig는 머신러닝의 엔지니어링 복잡성을 기본으로 처리하여, 연구자들이 가장 높은 수준의 추상화에서 모델 구축에 집중할 수 있게 합니다. `torch.nn.Module` 모델에 대한 데이터 전처리, 하이퍼파라미터 최적화, 디바이스 관리, 분산 학습이 완전히 무료로 제공됩니다.
+
+- **손쉬운 벤치마크 구축**
+
+  최신 기준 모델을 만들고 새 모델과 비교하는 것이 간단한 설정 변경만으로 가능합니다.
+
+- **새로운 아키텍처를 여러 문제와 데이터셋에 쉽게 적용**
+
+  Ludwig가 지원하는 광범위한 태스크 및 데이터셋 세트에 새 모델을 적용하세요. Ludwig에는 간단한 설정만으로 여러 데이터셋에서 여러 모델 실험을 실행할 수 있는 [전체 벤치마킹 도구](https://arxiv.org/abs/2111.04260)가 모든 사용자에게 제공됩니다.
+
+- **데이터 전처리, 모델링, 메트릭의 높은 설정 가능성**
+
+  모델 아키텍처, 학습 루프, 하이퍼파라미터 검색, 백엔드 인프라의 모든 측면을 선언적 설정에서 추가 필드로 수정하여 파이프라인을 요구 사항에 맞게 커스터마이즈할 수 있습니다. 설정 가능한 항목에 대한 자세한 내용은 [Ludwig 설정](https://ludwig.ai/latest/configuration/) 문서를 확인하세요.
+
+- **멀티모달, 멀티태스크 학습 기본 지원**
+
+  코드 작성 없이 테이블 데이터, 텍스트, 이미지, 오디오까지 복잡한 모델 설정으로 혼합하여 사용할 수 있습니다.
+
+- **풍부한 모델 내보내기 및 추적**
+
+  Tensorboard, Comet ML, Weights & Biases, MLFlow, Aim Stack 등의 도구로 모든 시도와 메트릭을 자동으로 추적합니다.
+
+- **멀티 GPU, 멀티 노드 클러스터로 학습 자동 확장**
+
+  로컬 머신에서 클라우드로 코드 변경 없이 전환할 수 있습니다.
+
+- **사전 학습된 Huggingface Transformers를 포함한 최신 모델의 로우코드 인터페이스**
+
+  Ludwig는 [Huggingface Transformers](https://huggingface.co/docs/transformers/index)에서 제공하는 사전 학습된 모델과 네이티브로 통합됩니다. 사용자는 코드를 전혀 작성하지 않고도 방대한 최신 사전 학습 PyTorch 모델을 사용할 수 있습니다. 예를 들어, Ludwig로 BERT 기반 감성 분석 모델을 학습시키는 것은 다음과 같이 간단합니다:
+
+  ```shell
+  ludwig train --dataset sst5 --config_str "{input_features: [{name: sentence, type: text, encoder: bert}], output_features: [{name: label, type: category}]}"
+  ```
+
+- **AutoML을 위한 로우코드 인터페이스**
+
+  [Ludwig AutoML](https://ludwig.ai/latest/user_guide/automl/)을 사용하면 데이터셋, 대상 컬럼, 시간 예산만 제공하여 학습된 모델을 얻을 수 있습니다.
+
+  ```python
+  auto_train_results = ludwig.automl.auto_train(dataset=my_dataset_df, target=target_column_name, time_limit_s=7200)
+  ```
+
+- **손쉬운 프로덕션화**
+
+  Ludwig는 GPU를 포함한 딥러닝 모델 서빙을 쉽게 만들어 줍니다. 학습된 Ludwig 모델에 대한 REST API를 실행하세요.
+
+  ```shell
+  ludwig serve --model_path=/path/to/model
+  ```
+
+  Ludwig는 효율적인 Torchscript 번들로 모델 내보내기를 지원합니다.
+
+  ```shell
+  ludwig export_torchscript --model_path=/path/to/model
+  ```
+
+# 📚 튜토리얼
+
+- [텍스트 분류](https://ludwig.ai/latest/examples/text_classification)
+- [테이블 데이터 분류](https://ludwig.ai/latest/examples/adult_census_income)
+- [이미지 분류](https://ludwig.ai/latest/examples/mnist)
+- [멀티모달 분류](https://ludwig.ai/latest/examples/multimodal_classification)
+
+# 🔬 예제 사용 사례
+
+- [개체명 인식 태깅](https://ludwig.ai/latest/examples/ner_tagging)
+- [자연어 이해](https://ludwig.ai/latest/examples/nlu)
+- [기계 번역](https://ludwig.ai/latest/examples/machine_translation)
+- [seq2seq를 통한 대화 모델링](https://ludwig.ai/latest/examples/seq2seq)
+- [감성 분석](https://ludwig.ai/latest/examples/sentiment_analysis)
+- [시아미즈 네트워크를 이용한 원샷 학습](https://ludwig.ai/latest/examples/oneshot)
+- [시각적 질의응답](https://ludwig.ai/latest/examples/visual_qa)
+- [음성 숫자 인식](https://ludwig.ai/latest/examples/speech_recognition)
+- [화자 인증](https://ludwig.ai/latest/examples/speaker_verification)
+- [이진 분류 (타이타닉)](https://ludwig.ai/latest/examples/titanic)
+- [시계열 예측](https://ludwig.ai/latest/examples/forecasting)
+- [시계열 예측 (날씨)](https://ludwig.ai/latest/examples/weather)
+- [영화 평점 예측](https://ludwig.ai/latest/examples/movie_ratings)
+- [다중 레이블 분류](https://ludwig.ai/latest/examples/multi_label)
+- [멀티태스크 학습](https://ludwig.ai/latest/examples/multi_task)
+- [단순 회귀: 연비 예측](https://ludwig.ai/latest/examples/fuel_efficiency)
+- [사기 탐지](https://ludwig.ai/latest/examples/fraud)
+
+# 💡 추가 정보
+
+[Ludwig](https://arxiv.org/pdf/1909.07930.pdf), [선언적 ML](https://arxiv.org/pdf/2107.08148.pdf), [Ludwig의 SoTA 벤치마크](https://openreview.net/pdf?id=hwjnu6qW7E4)에 대한 논문을 읽어보세요.
+
+[Ludwig의 작동 방식](https://ludwig.ai/latest/user_guide/how_ludwig_works/), [시작 가이드](https://ludwig.ai/latest/getting_started/), 더 많은 [예제](https://ludwig.ai/latest/examples)를 확인하세요.
+
+[기여](https://github.com/ludwig-ai/ludwig/blob/main/CONTRIBUTING.md)에 관심이 있으시거나, 질문, 의견, 공유하고 싶은 생각이 있으시거나, 최신 정보를 받고 싶으시다면 [Discord 커뮤니티에 참여](https://discord.gg/CBgdrGnZjy)하시고 [X](https://twitter.com/ludwig_ai)에서 팔로우해 주세요!
+
+# 🤝 함께 Ludwig를 만들어 갈 커뮤니티에 참여하세요
+
+Ludwig는 여러분과 같은 분들의 기여에 의존하는 활발하게 관리되는 오픈소스 프로젝트입니다. Ludwig를 모든 사람이 사용할 수 있는 더 접근 가능하고 기능이 풍부한 프레임워크로 만들기 위해 활발한 Ludwig 기여자 그룹에 참여하는 것을 고려해 주세요!
+
+<a href="https://github.com/ludwig-ai/ludwig/graphs/contributors">
+  <img src="https://contrib.rocks/image?repo=ludwig-ai/ludwig" />
+</a><br/>
+
+## Star History
+
+[![Star History Chart](https://api.star-history.com/svg?repos=ludwig-ai/ludwig&type=Date)](https://star-history.com/#ludwig-ai/ludwig&Date)
+
+# 👋 참여하기
+
+- [Discord](https://discord.gg/CBgdrGnZjy)
+- [X (Twitter)](https://twitter.com/ludwig_ai)
+- [Medium](https://medium.com/ludwig-ai)
+- [GitHub Issues](https://github.com/ludwig-ai/ludwig/issues)

RELEASES.md

@@ -0,0 +1,50 @@
+# Releasing
+
+## Release procedure
+
+1. Update version number in `ludwig/globals.py`
+1. Update the `README.md` file
+1. Update `ludwig-docs`
+1. Commit
+1. Tag the commit with the version number `vX.Y.Z` with a meaningful message
+1. Push with `--tags`
+1. If a non-patch release, edit the release notes
+1. The PyPI upload is automated via GitHub Actions (`.github/workflows/upload-pypi.yml`) when a release is published
+1. Publish Docker images (see below)
+
+## Docker images
+
+Four images are published to Docker Hub under the `ludwigai` organisation for each release:
+`ludwigai/ludwig`, `ludwigai/ludwig-gpu`, `ludwigai/ludwig-ray`, `ludwigai/ludwig-ray-gpu`.
+
+### Automated (CI)
+
+The GitHub Actions workflow `.github/workflows/docker.yml` triggers on `v*.*.*` tags and builds
+images from the tagged source. If CI is healthy this runs automatically after step 6 above.
+
+### Manual fallback
+
+If CI does not run or images need to be backfilled, trigger a versioned build via the workflow
+dispatch input — no local Docker setup required:
+
+```bash
+# Trigger all 4 image variants for a specific PyPI release
+gh workflow run docker.yml --repo ludwig-ai/ludwig --ref main \
+  -f ludwig_version=0.14.0 -f latest=true
+```
+
+Or build and push locally using the script at `docker/build_and_push.sh`
+(requires `docker login` to a `ludwigai` Docker Hub account):
+
+```bash
+./docker/build_and_push.sh 0.14.0 --latest
+```
+
+Both approaches install `ludwig[full]==<version>` from PyPI and produce two tags per image:
+the full version (`0.14.0`) and the major.minor shorthand (`0.14`), plus `latest` when requested.
+
+## Release policy
+
+Ludwig follows [Semantic Versioning](https://semver.org).
+In general, for major and minor releases, maintainers should all agree on the release.
+For patches, in particular time sensitive ones, a single maintainer can release without a full consensus, but this practice should be reserved for critical situations.

docker/README.md

@@ -0,0 +1,137 @@
+# Ludwig Docker Images
+
+These images provide Ludwig, a toolbox to train and evaluate deep learning models
+without the need to write code. Ludwig Docker images contain the full set of pre-requisite
+packages to support these capabilities
+
+- text features
+- image features
+- audio features
+- visualizations
+- hyperparameter optimization
+- distributed training
+- model serving
+
+## Publishing images
+
+Images are normally published automatically by CI (`.github/workflows/docker.yml`) when a release
+tag is pushed. To publish manually or backfill a release, use the script in this directory:
+
+```bash
+# Requires: docker login to a ludwigai Docker Hub account
+./docker/build_and_push.sh <version> [--latest]
+
+# Examples
+./docker/build_and_push.sh 0.14.0 --latest   # new latest release
+./docker/build_and_push.sh 0.13.0            # backfill without updating :latest
+```
+
+See `RELEASES.md` for the full release procedure.
+
+## Repositories
+
+These four repositories contain a version of Ludwig with full features built
+from the project's `master` branch.
+
+- `ludwigai/ludwig` Ludwig packaged with PyTorch
+- `ludwigai/ludwig-gpu` Ludwig packaged with gpu-enabled version of PyTorch
+- `ludwigai/ludwig-ray` Ludwig packaged with PyTorch
+  and Ray 2.3.1 (https://github.com/ray-project/ray)
+- `ludwigai/ludwig-ray-gpu` Ludwig packaged with gpu-enabled versions of PyTorch
+  and Ray 2.3.1 (https://github.com/ray-project/ray)
+
+## Image Tags
+
+- `master` - built from Ludwig's `master` branch
+- `nightly` - nightly build of Ludwig's software.
+- `sha-<commit point>` - version of Ludwig software at designated git sha1
+  7-character commit point.
+
+## Running Containers
+
+Examples of using the `ludwigai/ludwig:master` image to:
+
+- run the `ludwig cli` command or
+- run Python program containing Ludwig api or
+- view Ludwig results with Tensorboard
+
+For purposes of the examples assume this host directory structure
+
+```
+/top/level/directory/path/
+    data/
+        train.csv
+    src/
+        config.yaml
+        ludwig_api_program.py
+```
+
+### Run Ludwig CLI
+
+```
+# set shell variable to parent directory
+parent_path=/top/level/directory/path
+
+# invoke docker run command to execute the ludwig cli
+# map host directory ${parent_path}/data to container /data directory
+# map host directory ${parent_path}/src to container /src directory
+docker run -v ${parent_path}/data:/data  \
+    -v ${parent_path}/src:/src \
+    ludwigai/ludwig:master \
+    experiment --config /src/config.yaml \
+        --dataset /data/train.csv \
+        --output_directory /src/results
+```
+
+Experiment results can be found in host directory `/top/level/directory/path/src/results`
+
+### Run Python program using Ludwig APIs
+
+```
+# set shell variable to parent directory
+parent_path=/top/level/directory/path
+
+# invoke docker run command to execute Python interpreter
+# map host directory ${parent_path}/data to container /data directory
+# map host directory ${parent_path}/src to container /src directory
+# set current working directory to container /src directory
+# change default entrypoint from ludwig to python
+docker run  -v ${parent_path}/data:/data  \
+    -v ${parent_path}/src:/src \
+    -w /src \
+    --entrypoint python \
+    ludwigai/ludwig:master /src/ludwig_api_program.py
+```
+
+Ludwig results can be found in host
+directory `/top/level/directory/path/src/results`
+
+### View Ludwig Tensorboard results
+
+```
+# set shell variable to parent directory
+parent_path=/top/level/directory/path
+
+# invoke docker run command to execute Tensorboard
+# map host directory ${parent_path}/src to container /src directory
+# set up mapping from localhost port 6006 to container port 6006
+# change default entrypoint from ludwig to tensorboard
+# --logdir container location of tenorboard logs /src/results/<experiment_name>_<model_name>/model/logs
+# --bind_all Tensorboard serves on all public container interfaces
+docker run  -v ${parent_path}/src:/src \
+    -p 6006:6006 \
+    --entrypoint tensorboard \
+    ludwigai/ludwig:master \
+      --logdir /src/results/experiment_run/model/logs \
+      --bind_all
+```
+
+Point browser to `http://localhost:6006` to see Tensorboard dashboard.
+
+### Devcontainer
+
+If you want to contribute to Ludwig, you can setup a Docker container with all the dependencies
+installed as a full featured development environment. This can be done using devcontainers with VS Code:
+https://code.visualstudio.com/docs/devcontainers/containers
+
+You can find the `devcontainer.json` file within the top level `.devcontainer` folder.

docker/build_and_push.sh

@@ -0,0 +1,74 @@
+#!/usr/bin/env bash
+# Build and push versioned Ludwig Docker images to Docker Hub.
+#
+# Usage:
+#   ./docker/build_and_push.sh <version> [--latest]
+#
+# Examples:
+#   ./docker/build_and_push.sh 0.14.0           # tags: 0.14.0, 0.14
+#   ./docker/build_and_push.sh 0.14.0 --latest  # tags: 0.14.0, 0.14, latest
+#
+# Requires: docker login to ludwigai account already done.
+
+set -euo pipefail
+
+if [ $# -lt 1 ]; then
+  echo "Usage: $0 <version> [--latest]"
+  echo "  version  full version to build, e.g. 0.14.0"
+  echo "  --latest also tag as :latest"
+  exit 1
+fi
+
+VERSION="$1"
+IS_LATEST=false
+if [ "${2:-}" = "--latest" ]; then
+  IS_LATEST=true
+fi
+
+# Derive major.minor tag from full version (e.g. "0.14.0" -> "0.14")
+MINOR="${VERSION%.*}"
+
+REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
+
+# Image variants: "image_name dockerfile_dir"
+VARIANTS=(
+  "ludwigai/ludwig         docker/ludwig"
+  "ludwigai/ludwig-gpu     docker/ludwig-gpu"
+  "ludwigai/ludwig-ray     docker/ludwig-ray"
+  "ludwigai/ludwig-ray-gpu docker/ludwig-ray-gpu"
+)
+
+build_and_push() {
+  local image="$1"
+  local dockerfile_dir="$2"
+
+  echo ""
+  echo "=== Building ${image}:${VERSION} ==="
+
+  local tag_args="-t ${image}:${VERSION} -t ${image}:${MINOR}"
+  if [ "${IS_LATEST}" = "true" ]; then
+    tag_args="${tag_args} -t ${image}:latest"
+  fi
+
+  # shellcheck disable=SC2086
+  docker build \
+    --build-arg LUDWIG_VERSION="${VERSION}" \
+    ${tag_args} \
+    -f "${REPO_ROOT}/${dockerfile_dir}/Dockerfile" \
+    "${REPO_ROOT}"
+
+  echo "--- Pushing ${image}:${VERSION} ---"
+  docker push "${image}:${VERSION}"
+  docker push "${image}:${MINOR}"
+  if [ "${IS_LATEST}" = "true" ]; then
+    docker push "${image}:latest"
+  fi
+}
+
+for variant_entry in "${VARIANTS[@]}"; do
+  read -r image dockerfile_dir <<< "${variant_entry}"
+  build_and_push "${image}" "${dockerfile_dir}"
+done
+
+echo ""
+echo "All images built and pushed successfully."

docker/ludwig-gpu/Dockerfile

@@ -0,0 +1,43 @@
+#
+# Ludwig Docker image with full set of pre-requiste packages to support these capabilities
+#   text features
+#   image features
+#   audio features
+#   visualizations
+#   hyperparameter optimization
+#   distributed training
+#   model serving
+#
+
+FROM nvidia/cuda:12.6.3-cudnn-devel-ubuntu24.04
+
+RUN apt-get -y update && DEBIAN_FRONTEND="noninteractive" apt-get -y install \
+    python3.12 \
+    python3.12-venv \
+    python3.12-dev \
+    git \
+    libsndfile1 \
+    cmake \
+    ffmpeg \
+    sox \
+    libsox-dev
+
+RUN python3.12 -m venv /opt/ludwig-venv
+ENV PATH="/opt/ludwig-venv/bin:$PATH"
+RUN pip install -U pip
+
+ARG LUDWIG_VERSION
+
+WORKDIR /ludwig
+
+COPY . .
+RUN if [ -n "${LUDWIG_VERSION}" ]; then \
+      pip install --no-cache-dir "ludwig[full]==${LUDWIG_VERSION}" --extra-index-url https://download.pytorch.org/whl/cu126; \
+    else \
+      pip install --no-cache-dir '.[full]' --extra-index-url https://download.pytorch.org/whl/cu126; \
+    fi
+RUN pip install --no-cache-dir --force-reinstall torch==2.12.0 torchvision==0.27.0 torchaudio==2.11.0 --index-url https://download.pytorch.org/whl/cu126
+
+WORKDIR /data
+
+ENTRYPOINT ["ludwig"]

docker/ludwig-ray-gpu/Dockerfile

@@ -0,0 +1,40 @@
+#
+# Ludwig Docker image with Ray support and full dependencies including:
+#   text features
+#   image features
+#   audio features
+#   visualizations
+#   hyperparameter optimization
+#   distributed training
+#   model serving
+#
+
+FROM rayproject/ray:2.54.0-py312-cu126
+
+RUN sudo apt-get update && \
+    DEBIAN_FRONTEND="noninteractive" sudo apt-get install -y \
+    build-essential \
+    wget \
+    git \
+    curl \
+    libsndfile1 \
+    cmake \
+    tzdata \
+    rsync \
+    vim \
+    ffmpeg \
+    sox \
+    libsox-dev
+RUN pip install -U pip
+
+ARG LUDWIG_VERSION
+
+WORKDIR /ludwig
+
+COPY . .
+RUN if [ -n "${LUDWIG_VERSION}" ]; then \
+      pip install --no-cache-dir "ludwig[full]==${LUDWIG_VERSION}" --extra-index-url https://download.pytorch.org/whl/cu126; \
+    else \
+      pip install --no-cache-dir '.[full]' --extra-index-url https://download.pytorch.org/whl/cu126; \
+    fi
+RUN pip install --no-cache-dir --force-reinstall torch==2.12.0 torchvision==0.27.0 torchaudio==2.11.0 --index-url https://download.pytorch.org/whl/cu126

docker/ludwig-ray/Dockerfile

@@ -0,0 +1,39 @@
+#
+# Ludwig Docker image with Ray support and full dependencies including:
+#   text features
+#   image features
+#   audio features
+#   visualizations
+#   hyperparameter optimization
+#   distributed training
+#   model serving
+#
+
+FROM rayproject/ray:2.54.0-py312
+
+RUN sudo apt-get update && DEBIAN_FRONTEND="noninteractive" sudo apt-get install -y \
+	build-essential \
+	wget \
+	git \
+	curl \
+	libsndfile1 \
+	cmake \
+	tzdata \
+	rsync \
+	vim \
+	ffmpeg \
+	sox \
+	libsox-dev
+RUN pip install -U pip
+
+ARG LUDWIG_VERSION
+
+WORKDIR /ludwig
+
+COPY . .
+RUN if [ -n "${LUDWIG_VERSION}" ]; then \
+      pip install --no-cache-dir "ludwig[full]==${LUDWIG_VERSION}" --extra-index-url https://download.pytorch.org/whl/cpu; \
+    else \
+      pip install --no-cache-dir '.[full]' --extra-index-url https://download.pytorch.org/whl/cpu; \
+    fi
+RUN pip install --no-cache-dir --force-reinstall torch==2.12.0 torchvision==0.27.0 torchaudio==2.11.0 --extra-index-url https://download.pytorch.org/whl/cpu

docker/ludwig/Dockerfile

@@ -0,0 +1,39 @@
+#
+# Ludwig Docker image with full set of pre-requiste packages to support these capabilities
+#   text features
+#   image features
+#   audio features
+#   visualizations
+#   hyperparameter optimization
+#   distributed training
+#   model serving
+#
+
+FROM python:3.12-slim
+
+RUN apt-get -y update && apt-get -y install \
+    git \
+    libsndfile1 \
+    build-essential \
+    g++ \
+    cmake \
+    ffmpeg \
+    sox \
+    libsox-dev
+RUN pip install -U pip
+
+ARG LUDWIG_VERSION
+
+WORKDIR /ludwig
+
+COPY . .
+RUN if [ -n "${LUDWIG_VERSION}" ]; then \
+      pip install --no-cache-dir "ludwig[full]==${LUDWIG_VERSION}" --extra-index-url https://download.pytorch.org/whl/cpu; \
+    else \
+      pip install --no-cache-dir '.[full]' --extra-index-url https://download.pytorch.org/whl/cpu; \
+    fi
+RUN pip install --no-cache-dir --force-reinstall torch==2.12.0 torchvision==0.27.0 torchaudio==2.11.0 --extra-index-url https://download.pytorch.org/whl/cpu
+
+WORKDIR /data
+
+ENTRYPOINT ["ludwig"]

docs/developer_guide/adding_a_feature_type.md

@@ -0,0 +1,348 @@
+# Adding a New Feature Type to Ludwig
+
+This guide walks through every file you need to touch when adding a brand-new feature type (e.g. a hypothetical `"widget"` type). Use `ludwig/features/binary_feature.py` and `ludwig/schema/features/binary_feature.py` as living reference implementations — they are among the simplest complete examples.
+
+______________________________________________________________________
+
+## Conceptual overview
+
+Each feature type lives in two parallel places:
+
+| Layer              | Location                                   | Purpose                                                                       |
+| ------------------ | ------------------------------------------ | ----------------------------------------------------------------------------- |
+| **Schema**         | `ludwig/schema/features/<type>_feature.py` | Pydantic-backed config classes; declares hyperparameters and their defaults   |
+| **Feature module** | `ludwig/features/<type>_feature.py`        | PyTorch modules; implements preprocessing, encoding, decoding, postprocessing |
+
+The schema classes are used for config validation and serialization. The feature module classes are instantiated at model-build time using those configs. Neither layer knows the other exists at import time — they are wired together through the feature registry.
+
+______________________________________________________________________
+
+## Step 1 ��� Define the constant
+
+Add the type string to `ludwig/constants.py`:
+
+```python
+WIDGET = "widget"
+```
+
+______________________________________________________________________
+
+## Step 2 — Write the schema file
+
+Create `ludwig/schema/features/widget_feature.py`. The minimal required structure is:
+
+```python
+from ludwig.constants import WIDGET, MODEL_ECD
+from ludwig.schema import utils as schema_utils
+from ludwig.schema.encoders.base import BaseEncoderConfig
+from ludwig.schema.encoders.utils import EncoderDataclassField
+from ludwig.schema.features.base import BaseInputFeatureConfig, BaseOutputFeatureConfig
+from ludwig.schema.features.preprocessing.base import BasePreprocessingConfig
+from ludwig.schema.features.preprocessing.utils import PreprocessingDataclassField
+from ludwig.schema.features.utils import (
+    ecd_defaults_config_registry,
+    ecd_input_config_registry,
+    ecd_output_config_registry,
+    input_mixin_registry,
+    output_mixin_registry,
+)
+from ludwig.schema.utils import LudwigBaseConfig
+
+
+@input_mixin_registry.register(WIDGET)
+class WidgetInputFeatureConfigMixin(LudwigBaseConfig):
+    preprocessing: BasePreprocessingConfig = PreprocessingDataclassField(feature_type=WIDGET)
+
+
+class WidgetInputFeatureConfig(WidgetInputFeatureConfigMixin, BaseInputFeatureConfig):
+    type: str = schema_utils.ProtectedString(WIDGET)
+    encoder: BaseEncoderConfig = None
+
+
+@ecd_input_config_registry.register(WIDGET)
+class ECDWidgetInputFeatureConfig(WidgetInputFeatureConfig):
+    encoder: BaseEncoderConfig = EncoderDataclassField(
+        MODEL_ECD,
+        feature_type=WIDGET,
+        default="dense",  # default encoder for this type
+    )
+
+
+# For output features only:
+@output_mixin_registry.register(WIDGET)
+class WidgetOutputFeatureConfigMixin(LudwigBaseConfig):
+    # add loss, calibration, etc. fields here
+    pass
+
+
+class WidgetOutputFeatureConfig(WidgetOutputFeatureConfigMixin, BaseOutputFeatureConfig):
+    type: str = schema_utils.ProtectedString(WIDGET)
+    default_validation_metric: str = "some_metric"
+
+
+@ecd_output_config_registry.register(WIDGET)
+class ECDWidgetOutputFeatureConfig(WidgetOutputFeatureConfig):
+    pass
+```
+
+**Key rules:**
+
+- `type` must be a `ProtectedString` with your constant — this prevents accidental overwrite via user YAML.
+- `@input_mixin_registry.register` / `@output_mixin_registry.register` make the preprocessing config available to `global_defaults` in Ludwig configs.
+- `@ecd_input_config_registry.register` / `@ecd_output_config_registry.register` wire the schema into the ECD model config builder.
+
+______________________________________________________________________
+
+## Step 3 — Write the preprocessing config
+
+Create `ludwig/schema/features/preprocessing/widget_feature_preprocessing.py` if your feature needs non-default preprocessing parameters, or register your type against an existing one (e.g. `number_feature` for scalars). For a new type, create the file:
+
+```python
+from ludwig.schema.features.preprocessing.base import BasePreprocessingConfig
+from ludwig.schema.features.preprocessing.utils import register_preprocessor
+from ludwig.constants import WIDGET
+
+
+@register_preprocessor(WIDGET)
+class WidgetPreprocessingConfig(BasePreprocessingConfig):
+    # add preprocessing hyperparameters here
+    pass
+```
+
+______________________________________________________________________
+
+## Step 4 — Write the feature module
+
+Create `ludwig/features/widget_feature.py`. The required classes are:
+
+### Inner preprocessing module
+
+```python
+import torch
+from ludwig.features.base_feature import BasePreprocessingModule, FeaturePreprocessingMixin, InputFeature, OutputFeature
+
+
+class _WidgetPreprocessing(BasePreprocessingModule):
+    """Runs inside the model graph during inference to preprocess raw input."""
+
+    def __init__(self, metadata: dict, preprocessing_config, is_input_feature: bool = True):
+        super().__init__()
+        # store everything needed to preprocess at inference time
+
+    def forward(self, v):
+        # v is the raw column value; return a tensor
+        raise NotImplementedError
+```
+
+### FeatureMixin (shared preprocessing logic)
+
+`FeaturePreprocessingMixin` provides the Python-side preprocessing used during dataset preparation (not inside the model graph). You must implement `add_feature_data` and `get_preprocessing_module`:
+
+```python
+class WidgetFeatureMixin(FeaturePreprocessingMixin):
+    @staticmethod
+    def type():
+        return WIDGET
+
+    @staticmethod
+    def cast_column(column, backend):
+        """Cast the raw DataFrame column to the expected dtype."""
+        return column
+
+    @staticmethod
+    def add_feature_data(
+        feature_config,
+        input_df,
+        proc_df,
+        metadata,
+        preprocessing_parameters,
+        backend,
+        skip_save_processed_input,
+    ):
+        """Populate proc_df[feature_config[PROC_COLUMN]] with preprocessed values."""
+        proc_df[feature_config[PROC_COLUMN]] = input_df[feature_config[COLUMN]].values
+        return proc_df
+
+    @staticmethod
+    def fill_missing_values(feature_config, input_df, backend):
+        """Replace NaN/None with a fill value appropriate for this type."""
+        return input_df
+
+    @staticmethod
+    def feature_meta(column, preprocessing_parameters, backend):
+        """Compute and return the training-set-level metadata dict for this feature."""
+        return {}
+
+    @staticmethod
+    def get_preprocessing_module(feature_config, metadata):
+        """Return the _WidgetPreprocessing module for use during inference."""
+        return _WidgetPreprocessing(metadata, feature_config.preprocessing)
+```
+
+### InputFeature class
+
+```python
+from ludwig.schema.features.widget_feature import WidgetInputFeatureConfig
+
+
+class WidgetInputFeature(WidgetFeatureMixin, InputFeature):
+    def __init__(self, input_feature_config: WidgetInputFeatureConfig, encoder_obj=None, **kwargs):
+        super().__init__(input_feature_config, **kwargs)
+        self._input_shape = torch.Size([1])  # set to actual encoded shape
+
+        if encoder_obj:
+            self.encoder_obj = encoder_obj
+        else:
+            self.encoder_obj = self.initialize_encoder(input_feature_config.encoder)
+
+    def forward(self, inputs, mask=None):
+        assert inputs.dtype == torch.float32
+        encoder_output = self.encoder_obj(inputs, mask=mask)
+        return {"encoder_output": encoder_output}
+
+    @property
+    def input_dtype(self):
+        return torch.float32
+
+    @property
+    def input_shape(self):
+        return self._input_shape
+
+    @property
+    def output_shape(self):
+        return self.encoder_obj.output_shape
+
+    @staticmethod
+    def update_config_with_metadata(feature_config, feature_metadata, *args, **kwargs):
+        pass
+
+    @staticmethod
+    def create_sample_input(batch_size=2):
+        return torch.zeros(batch_size, 1)
+
+    @staticmethod
+    def get_schema_cls():
+        return WidgetInputFeatureConfig
+```
+
+### OutputFeature class (only if this type can be a target)
+
+```python
+from ludwig.schema.features.widget_feature import WidgetOutputFeatureConfig
+
+
+class WidgetOutputFeature(WidgetFeatureMixin, OutputFeature):
+    def __init__(self, output_feature_config: WidgetOutputFeatureConfig, output_features: dict, **kwargs):
+        super().__init__(output_feature_config, output_features, **kwargs)
+        self._input_shape = torch.Size([output_feature_config.input_size])
+        self.decoder_obj = self.initialize_decoder(output_feature_config.decoder)
+        self._setup_loss()
+        self._setup_metrics()
+
+    def logits(self, inputs, target=None):
+        return self.decoder_obj(inputs)
+
+    def create_predict_module(self):
+        return _WidgetPredict()  # see PredictModule below
+
+    def get_prediction_set(self):
+        return {LOGITS, PREDICTIONS, PROBABILITIES}
+
+    @classmethod
+    def update_config_with_metadata(cls, feature_config, feature_metadata, *args, **kwargs):
+        feature_config.input_size = feature_metadata["input_size"]
+
+    @staticmethod
+    def get_schema_cls():
+        return WidgetOutputFeatureConfig
+```
+
+### PredictModule (for output features)
+
+```python
+from ludwig.features.base_feature import PredictModule
+
+
+class _WidgetPredict(PredictModule):
+    def forward(self, inputs, feature_name):
+        logits = inputs[f"{feature_name}_{LOGITS}"]
+        predictions = (logits > 0.5).float()
+        return {PREDICTIONS: predictions, LOGITS: logits}
+```
+
+______________________________________________________________________
+
+## Step 5 — Register in the feature registries
+
+Open `ludwig/features/feature_registries.py` and add your classes to all relevant registry functions:
+
+```python
+# at the top — add import
+from ludwig.features.widget_feature import WidgetFeatureMixin, WidgetInputFeature
+
+# in get_base_type_registry(), inside the returned dict:
+#     WIDGET: WidgetFeatureMixin,
+#
+# in get_input_type_registry(), inside the returned dict:
+#     WIDGET: WidgetInputFeature,
+#
+# in get_output_type_registry() if applicable, inside the returned dict:
+#     WIDGET: WidgetOutputFeature,
+```
+
+The model builder uses `get_input_type_registry()` and `get_output_type_registry()` to instantiate feature objects from config at training time.
+
+______________________________________________________________________
+
+## Step 6 — Register the constant in constants.py (feature sets)
+
+If the feature appears in `FEATURE_TYPES`, `INPUT_FEATURE_TYPES`, or similar sets, add `WIDGET` there too.
+
+______________________________________________________________________
+
+## Step 7 — Write tests
+
+Create `tests/ludwig/features/test_widget_feature.py`. At minimum test:
+
+1. `WidgetFeatureMixin.add_feature_data` — correct column values written to `proc_df`
+1. `_WidgetPreprocessing.forward` — correct tensor shape for a known input
+1. `WidgetInputFeature.forward` — correct output keys and shapes with a random input
+1. Encoder round-trip via `create_sample_input`
+
+```python
+import torch
+import pytest
+from tests.integration_tests.utils import generate_data, run_api_test
+
+
+def test_widget_preprocessing_forward():
+    meta = {}
+    module = _WidgetPreprocessing(meta, preprocessing_config=None)
+    out = module(torch.zeros(4))
+    assert out.shape == (4, 1)
+```
+
+______________________________________________________________________
+
+## Checklist
+
+- [ ] `ludwig/constants.py` — add `WIDGET = "widget"`
+- [ ] `ludwig/schema/features/widget_feature.py` — schema classes + registry decorators
+- [ ] `ludwig/schema/features/preprocessing/` — preprocessing config class (or reuse existing)
+- [ ] `ludwig/features/widget_feature.py` — preprocessing module, mixin, input/output feature classes
+- [ ] `ludwig/features/feature_registries.py` — add to `get_base_type_registry`, `get_input_type_registry`, optionally `get_output_type_registry`
+- [ ] `tests/ludwig/features/test_widget_feature.py` — unit tests for preprocessing and forward pass
+
+______________________________________________________________________
+
+## Common pitfalls
+
+**`proc_df[PROC_COLUMN]` vs `proc_df[COLUMN]`** — always write to `PROC_COLUMN` (the internal column name), not `COLUMN` (the raw user column name). They can differ when the user renames features.
+
+**`get_preprocessing_module` vs `add_feature_data`** — `add_feature_data` runs in Python at dataset preparation time (CPU, pandas). `get_preprocessing_module` returns a `torch.nn.Module` that runs inside the model graph at inference time. Both must produce compatible representations.
+
+**`input_shape` vs `output_shape`** — `InputFeature.input_shape` is the shape of the *raw preprocessed* tensor going into the encoder. `InputFeature.output_shape` is the encoder's output shape that feeds into the combiner. Return `self.encoder_obj.output_shape` for the latter.
+
+**Registry order matters** — the registry in `feature_registries.py` is read at import time. If you import your feature class before `feature_registries.py` is loaded, the registry will be empty. The correct order is always: define constants → define schema → define feature → add to registry.
+
+**Schema `type` field** — always use `schema_utils.ProtectedString(WIDGET)` not `str = WIDGET`. The protected string raises an error if a user tries to override it in their config YAML, which prevents subtle type mismatches.

examples/README.md

@@ -0,0 +1,13 @@
+# Examples
+
+This directory contains example programs demonstrating Ludwig's Python APIs.
+
+| Directory          | Examples Provided                                                                                                                                                                                                                                                                                                               |
+| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| hyperopt           | Demonstrates Ludwig's to hyper-parameter optimization capability.                                                                                                                                                                                                                                                               |
+| kfold_cv           | Provides two examples for performing a k-fold cross validation analysis. One example uses the `ludwig experiment` cli. The other example uses the `ludwig.experiment.kfold_cross_validate()` api function.                                                                                                                      |
+| mnist              | Creates a model config data structure from a yaml file and trains a model. Programmatically modify the model config data structure to evaluate several different neural network architectures. Jupyter notebook demonstrates using a hold-out test data set to visualize model performance for alternative model architectures. |
+| titanic            | Trains a simple model with model config contained in a yaml file. Trains multiple models from yaml files and generate visualizations to compare training results. Jupyter notebook demonstrating how to programmatically create visualizations.                                                                                 |
+| serve              | Demonstrates running Ludwig http model server. A sample Python program illustrates how to invoke the REST API to get predictions from input features.                                                                                                                                                                           |
+| class_imbalance    | Demonstrates using our class balancing feature to over-sample an imbalanced dataset.                                                                                                                                                                                                                                            |
+| ray/job_submission | Submit Ludwig training to a remote Ray cluster via Ray Job Submission. Avoids Ray Client issues with ray.data. Works with KubeRay, Anyscale, or any Ray cluster.                                                                                                                                                                |

examples/alignment/README.md

@@ -0,0 +1,113 @@
+# LLM Alignment with DPO and KTO
+
+[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ludwig-ai/ludwig/blob/main/examples/alignment/alignment_dpo.ipynb)
+
+This example shows how to align a large language model with human preferences using Ludwig's
+built-in preference learning trainers. Alignment training is typically applied after an initial
+supervised fine-tuning (SFT) stage to improve response quality, reduce harmful outputs, and teach
+the model to follow instructions more reliably.
+
+## What is alignment?
+
+Alignment refers to the process of shaping a model's behaviour to match human values and preferences.
+The classic approach — Reinforcement Learning from Human Feedback (RLHF) — requires training a
+separate reward model on human-ranked responses, then running a full RL loop (PPO) against it.
+Modern preference learning methods like DPO bypass the reward model entirely, making alignment
+cheaper and more stable to train.
+
+## When to use each trainer
+
+| Trainer | Data format                          | Use case                                                                     | Compute                                                                  |
+| ------- | ------------------------------------ | ---------------------------------------------------------------------------- | ------------------------------------------------------------------------ |
+| `dpo`   | `prompt`, `chosen`, `rejected`       | Preference pairs from human feedback; most widely studied                    | Medium — requires forward passes through both policy and reference model |
+| `kto`   | `prompt`, `response`, `label` (bool) | Single-label feedback (thumbs up/down); no paired responses needed           | Low — simpler loss than DPO                                              |
+| `orpo`  | `prompt`, `chosen`, `rejected`       | Single-stage SFT + alignment; no separate reference model                    | Low — no reference model forward passes                                  |
+| `grpo`  | `prompt`, custom reward function     | RL-style training with a group-normalised reward signal; used in DeepSeek-R1 | High — requires multiple rollouts per prompt                             |
+
+Choose **DPO** when you have human-ranked response pairs and want the best-studied approach.
+Choose **KTO** when collecting binary per-response feedback is easier than pairwise comparisons.
+Choose **ORPO** when you want to skip the SFT stage and align in one shot.
+Choose **GRPO** when you have a programmatic reward function (e.g. code execution, math verification).
+
+## Prerequisites
+
+- GPU with at least 40 GiB of VRAM (A100 recommended)
+- [HuggingFace API Token](https://huggingface.co/docs/hub/security-tokens)
+- Access approval to [Llama-3.1-8B](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B)
+
+## Quick start
+
+Install dependencies:
+
+```bash
+pip install "ludwig[llm]" datasets
+```
+
+Set your HuggingFace token:
+
+```bash
+export HUGGING_FACE_HUB_TOKEN="<your_token>"
+```
+
+Prepare the dataset:
+
+```bash
+python prepare_dataset.py
+```
+
+Run DPO training:
+
+```bash
+python train_dpo.py
+# or with the CLI:
+ludwig train --config config_dpo.yaml --dataset train.csv
+```
+
+Run KTO training:
+
+```bash
+ludwig train --config config_kto.yaml --dataset train_kto.csv
+```
+
+Run GRPO training (reuses the DPO preference-pair format):
+
+```bash
+python train_grpo.py
+# or with the CLI:
+ludwig train --config config_grpo.yaml --dataset preference_data.parquet
+```
+
+## GRPO specifics
+
+GRPO (Group Relative Policy Optimization, Shao et al. 2024) is the alignment method used by
+DeepSeek-R1. For each prompt it samples a group of `grpo_num_generations` completions, scores
+them, normalises rewards within the group, and applies a PPO-style clipped objective —
+without a separate critic model.
+
+Ludwig's GRPO trainer consumes the same `prompt` / `chosen` / `rejected` columns as DPO, so
+a programmatic reward function is implemented as a pre-processing step: score each candidate
+completion in your dataset preparation pipeline, then emit the top-scoring completion as
+`chosen` and the lowest as `rejected`. See `config_grpo.yaml` for the full list of knobs
+(`grpo_beta` for the KL penalty, `grpo_epsilon` for PPO clipping,
+`grpo_num_generations` for the group size).
+
+## Files
+
+| File                  | Description                                                        |
+| --------------------- | ------------------------------------------------------------------ |
+| `prepare_dataset.py`  | Downloads Anthropic/hh-rlhf and converts it to Ludwig format       |
+| `train_dpo.py`        | DPO training script using the Python API                           |
+| `train_grpo.py`       | GRPO training script using the Python API                          |
+| `config_dpo.yaml`     | Ludwig config for DPO                                              |
+| `config_kto.yaml`     | Ludwig config for KTO                                              |
+| `config_orpo.yaml`    | Ludwig config for ORPO                                             |
+| `config_grpo.yaml`    | Ludwig config for GRPO                                             |
+| `alignment_dpo.ipynb` | Colab-compatible notebook covering DPO, KTO evaluation, and upload |
+
+## Upload to HuggingFace
+
+After training, upload the aligned model:
+
+```bash
+ludwig upload hf_hub -r <your_org>/<model_name> -m results/experiment_run/model
+```

examples/alignment/alignment_dpo.ipynb

@@ -0,0 +1,477 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# LLM Alignment with DPO and KTO\n",
+    "\n",
+    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ludwig-ai/ludwig/blob/main/examples/alignment/alignment_dpo.ipynb)\n",
+    "\n",
+    "This notebook walks through aligning **Llama-3.1-8B** with human preferences using Ludwig's built-in\n",
+    "**Direct Preference Optimization (DPO)** trainer, then shows how to switch to **KTO** when paired\n",
+    "preference data is not available.\n",
+    "\n",
+    "We use the publicly available [Anthropic HH-RLHF](https://huggingface.co/datasets/Anthropic/hh-rlhf)\n",
+    "dataset, which contains helpfulness and harmlessness preference pairs collected from human annotators.\n",
+    "\n",
+    "### What you will learn\n",
+    "- How to parse the HH-RLHF dataset into Ludwig's expected column format\n",
+    "- How to configure and run DPO alignment training with LoRA\n",
+    "- How to evaluate response quality before and after alignment\n",
+    "- How to switch to KTO with a one-line config change\n",
+    "- How to upload the aligned model to HuggingFace Hub\n",
+    "\n",
+    "### Prerequisites\n",
+    "- **GPU**: A100 (40 GiB) recommended. T4 will work with smaller batch sizes but is slower.\n",
+    "- **HuggingFace token**: Required for Llama-3.1-8B. Set `HUGGING_FACE_HUB_TOKEN` before running.\n",
+    "- **Model access**: Request access at https://huggingface.co/meta-llama/Meta-Llama-3.1-8B"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Install dependencies"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!pip install \"ludwig[llm]\" datasets --quiet"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Setup"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "\n",
+    "# Set your HuggingFace token — required to download Llama-3.1-8B.\n",
+    "# In Colab: Secrets panel (key icon) -> add HF_TOKEN, then reference it here.\n",
+    "try:\n",
+    "    from google.colab import userdata\n",
+    "\n",
+    "    os.environ[\"HUGGING_FACE_HUB_TOKEN\"] = userdata.get(\"HF_TOKEN\")\n",
+    "except Exception:\n",
+    "    pass  # Running locally — export HUGGING_FACE_HUB_TOKEN in your shell instead\n",
+    "\n",
+    "assert os.environ.get(\"HUGGING_FACE_HUB_TOKEN\"), (\n",
+    "    \"HUGGING_FACE_HUB_TOKEN is not set. Add it to Colab Secrets or export it in your shell.\"\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import subprocess\n",
+    "\n",
+    "# Verify GPU availability\n",
+    "result = subprocess.run(\n",
+    "    [\"nvidia-smi\", \"--query-gpu=name,memory.total\", \"--format=csv,noheader\"], capture_output=True, text=True\n",
+    ")\n",
+    "if result.returncode == 0:\n",
+    "    print(\"GPU(s) detected:\")\n",
+    "    print(result.stdout.strip())\n",
+    "else:\n",
+    "    print(\"WARNING: No GPU detected. Training will be very slow on CPU.\")\n",
+    "    print(\"In Colab: Runtime -> Change runtime type -> A100 GPU\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Prepare dataset\n",
+    "\n",
+    "The [Anthropic HH-RLHF](https://huggingface.co/datasets/Anthropic/hh-rlhf) dataset stores full\n",
+    "multi-turn conversations in two columns: `chosen` (preferred response) and `rejected` (dispreferred\n",
+    "response). Each value is a raw conversation string like:\n",
+    "\n",
+    "```\n",
+    "\\n\\nHuman: How do I make pasta?\\n\\nAssistant: Boil water, add pasta...\\n\\nHuman: What sauce?\\n\\nAssistant: Try marinara...\n",
+    "```\n",
+    "\n",
+    "Ludwig's DPO trainer expects three columns: `prompt`, `chosen`, `rejected`, where `prompt` is the\n",
+    "last human turn and `chosen`/`rejected` are the final assistant responses.\n",
+    "\n",
+    "The cell below does that extraction."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import re\n",
+    "\n",
+    "import pandas as pd\n",
+    "from datasets import load_dataset\n",
+    "\n",
+    "\n",
+    "def extract_last_human_turn(conversation: str) -> str:\n",
+    "    \"\"\"Extract the last Human turn from an HH-RLHF conversation string.\"\"\"\n",
+    "    turns = re.findall(r\"\\n\\nHuman: (.*?)(?=\\n\\nAssistant:|\\Z)\", conversation, re.DOTALL)\n",
+    "    return turns[-1].strip() if turns else conversation.strip()\n",
+    "\n",
+    "\n",
+    "def extract_last_assistant_turn(conversation: str) -> str:\n",
+    "    \"\"\"Extract the last Assistant turn from an HH-RLHF conversation string.\"\"\"\n",
+    "    turns = re.findall(r\"\\n\\nAssistant: (.*?)(?=\\n\\nHuman:|\\Z)\", conversation, re.DOTALL)\n",
+    "    return turns[-1].strip() if turns else \"\"\n",
+    "\n",
+    "\n",
+    "def prepare_dpo_dataset(split, max_samples=None):\n",
+    "    if max_samples:\n",
+    "        split = split.select(range(min(max_samples, len(split))))\n",
+    "    rows = []\n",
+    "    for ex in split:\n",
+    "        prompt = extract_last_human_turn(ex[\"chosen\"])\n",
+    "        chosen = extract_last_assistant_turn(ex[\"chosen\"])\n",
+    "        rejected = extract_last_assistant_turn(ex[\"rejected\"])\n",
+    "        if prompt and chosen and rejected:\n",
+    "            rows.append({\"prompt\": prompt, \"chosen\": chosen, \"rejected\": rejected})\n",
+    "    return pd.DataFrame(rows)\n",
+    "\n",
+    "\n",
+    "print(\"Downloading Anthropic/hh-rlhf ...\")\n",
+    "hh = load_dataset(\"Anthropic/hh-rlhf\")\n",
+    "print(f\"Train size: {len(hh['train'])}, Test size: {len(hh['test'])}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Use a subset for a quick experiment. Remove max_samples caps for full training.\n",
+    "MAX_TRAIN = 5000\n",
+    "MAX_TEST = 500\n",
+    "\n",
+    "train_df = prepare_dpo_dataset(hh[\"train\"], max_samples=MAX_TRAIN)\n",
+    "test_df = prepare_dpo_dataset(hh[\"test\"], max_samples=MAX_TEST)\n",
+    "\n",
+    "train_df.to_csv(\"train.csv\", index=False)\n",
+    "test_df.to_csv(\"test.csv\", index=False)\n",
+    "\n",
+    "print(f\"Saved {len(train_df)} train rows to train.csv\")\n",
+    "print(f\"Saved {len(test_df)} test rows to test.csv\")\n",
+    "train_df.head(2)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Preview: show a prompt with its chosen vs rejected response\n",
+    "row = train_df.iloc[0]\n",
+    "print(\"=== PROMPT ===\")\n",
+    "print(row[\"prompt\"])\n",
+    "print(\"\\n=== CHOSEN ===\")\n",
+    "print(row[\"chosen\"])\n",
+    "print(\"\\n=== REJECTED ===\")\n",
+    "print(row[\"rejected\"])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## DPO training\n",
+    "\n",
+    "Direct Preference Optimization (DPO) treats alignment as a classification problem: given a prompt,\n",
+    "prefer `chosen` over `rejected`. The loss function directly optimises the log-ratio between the\n",
+    "policy's probability of the chosen response and the reference model's, without needing an explicit\n",
+    "reward model.\n",
+    "\n",
+    "Key hyperparameters:\n",
+    "- `beta`: KL penalty coefficient (0.1 is a typical starting point). Higher values keep the aligned\n",
+    "  model closer to the base model.\n",
+    "- `learning_rate`: 5e-7 is much lower than SFT — the policy already knows how to generate text;\n",
+    "  we're only nudging its preferences.\n",
+    "- `lora r=16, alpha=32`: Standard LoRA settings. Increase `r` for larger models or more complex tasks."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import logging\n",
+    "\n",
+    "import yaml\n",
+    "\n",
+    "from ludwig.api import LudwigModel\n",
+    "\n",
+    "dpo_config = yaml.safe_load(\"\"\"\n",
+    "model_type: llm\n",
+    "base_model: meta-llama/Llama-3.1-8B\n",
+    "\n",
+    "adapter:\n",
+    "  type: lora\n",
+    "  r: 16\n",
+    "  alpha: 32\n",
+    "  dropout: 0.05\n",
+    "\n",
+    "trainer:\n",
+    "  type: dpo\n",
+    "  epochs: 1\n",
+    "  learning_rate: 5.0e-7\n",
+    "  batch_size: 2\n",
+    "  gradient_accumulation_steps: 8\n",
+    "  beta: 0.1\n",
+    "\n",
+    "input_features:\n",
+    "  - name: prompt\n",
+    "    type: text\n",
+    "\n",
+    "output_features:\n",
+    "  - name: chosen\n",
+    "    type: text\n",
+    "\n",
+    "backend:\n",
+    "  type: local\n",
+    "\"\"\")\n",
+    "\n",
+    "dpo_model = LudwigModel(config=dpo_config, logging_level=logging.INFO)\n",
+    "\n",
+    "train_stats, _, output_directory = dpo_model.train(\n",
+    "    dataset=\"train.csv\",\n",
+    "    experiment_name=\"hh_rlhf_dpo\",\n",
+    "    output_directory=\"results\",\n",
+    ")\n",
+    "\n",
+    "print(f\"\\nModel saved to: {output_directory}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Evaluate\n",
+    "\n",
+    "We compare responses from the base model (no alignment) and the DPO-aligned model on a few\n",
+    "prompts from the test set. You should see the aligned model give more helpful, less evasive answers."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Load the base model for comparison\n",
+    "base_config = yaml.safe_load(\"\"\"\n",
+    "model_type: llm\n",
+    "base_model: meta-llama/Llama-3.1-8B\n",
+    "\n",
+    "input_features:\n",
+    "  - name: prompt\n",
+    "    type: text\n",
+    "\n",
+    "output_features:\n",
+    "  - name: chosen\n",
+    "    type: text\n",
+    "\n",
+    "backend:\n",
+    "  type: local\n",
+    "\"\"\")\n",
+    "\n",
+    "base_model = LudwigModel(config=base_config, logging_level=logging.WARNING)\n",
+    "_ = base_model.train(dataset=\"train.csv\", experiment_name=\"baseline\", output_directory=\"results_base\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "eval_prompts = test_df[\"prompt\"].head(3).tolist()\n",
+    "eval_df = pd.DataFrame({\"prompt\": eval_prompts})\n",
+    "\n",
+    "base_preds, _ = base_model.predict(dataset=eval_df)\n",
+    "dpo_preds, _ = dpo_model.predict(dataset=eval_df)\n",
+    "\n",
+    "for i, prompt in enumerate(eval_prompts):\n",
+    "    print(f\"\\n{'=' * 60}\")\n",
+    "    print(f\"PROMPT: {prompt}\")\n",
+    "    print(\"\\n--- Base model ---\")\n",
+    "    print(base_preds.iloc[i][\"chosen_predictions\"])\n",
+    "    print(\"\\n--- DPO-aligned model ---\")\n",
+    "    print(dpo_preds.iloc[i][\"chosen_predictions\"])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## KTO alternative\n",
+    "\n",
+    "**Kahneman-Tversky Optimization (KTO)** is a good alternative to DPO when you cannot collect\n",
+    "paired preference data. Instead of a (chosen, rejected) pair per prompt, KTO requires only a\n",
+    "single response with a boolean label indicating whether it was desirable.\n",
+    "\n",
+    "This makes it easy to use binary user feedback (thumbs up / thumbs down, click-through, etc.)\n",
+    "as training signal.\n",
+    "\n",
+    "### Dataset format for KTO\n",
+    "\n",
+    "We expand each DPO pair into two rows:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "rows_kto = []\n",
+    "for _, row in train_df.iterrows():\n",
+    "    rows_kto.append({\"prompt\": row[\"prompt\"], \"response\": row[\"chosen\"], \"label\": True})\n",
+    "    rows_kto.append({\"prompt\": row[\"prompt\"], \"response\": row[\"rejected\"], \"label\": False})\n",
+    "\n",
+    "train_kto_df = pd.DataFrame(rows_kto)\n",
+    "train_kto_df.to_csv(\"train_kto.csv\", index=False)\n",
+    "print(f\"KTO dataset: {len(train_kto_df)} rows\")\n",
+    "train_kto_df.head(4)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "kto_config = yaml.safe_load(\"\"\"\n",
+    "model_type: llm\n",
+    "base_model: meta-llama/Llama-3.1-8B\n",
+    "\n",
+    "adapter:\n",
+    "  type: lora\n",
+    "  r: 16\n",
+    "  alpha: 32\n",
+    "  dropout: 0.05\n",
+    "\n",
+    "trainer:\n",
+    "  type: kto\n",
+    "  epochs: 1\n",
+    "  learning_rate: 5.0e-7\n",
+    "  batch_size: 2\n",
+    "  gradient_accumulation_steps: 8\n",
+    "  beta: 0.1\n",
+    "  desirable_weight: 1.0\n",
+    "  undesirable_weight: 1.0\n",
+    "\n",
+    "input_features:\n",
+    "  - name: prompt\n",
+    "    type: text\n",
+    "\n",
+    "output_features:\n",
+    "  - name: response\n",
+    "    type: text\n",
+    "\n",
+    "backend:\n",
+    "  type: local\n",
+    "\"\"\")\n",
+    "\n",
+    "kto_model = LudwigModel(config=kto_config, logging_level=logging.INFO)\n",
+    "\n",
+    "_, _, kto_output_dir = kto_model.train(\n",
+    "    dataset=\"train_kto.csv\",\n",
+    "    experiment_name=\"hh_rlhf_kto\",\n",
+    "    output_directory=\"results_kto\",\n",
+    ")\n",
+    "\n",
+    "print(f\"\\nKTO model saved to: {kto_output_dir}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Save and upload\n",
+    "\n",
+    "After training, upload the aligned model to HuggingFace Hub to share it or deploy it to an endpoint."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Replace with your HuggingFace org/username and desired repo name\n",
+    "HF_REPO = \"your-username/llama-3.1-8b-dpo-hh-rlhf\"\n",
+    "\n",
+    "# Upload via CLI:\n",
+    "print(f\"ludwig upload hf_hub -r {HF_REPO} -m {output_directory}\")\n",
+    "\n",
+    "# Or via Python API:\n",
+    "# LudwigModel.upload_to_hf_hub(HF_REPO, output_directory)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Uncomment to actually run the upload:\n",
+    "# !ludwig upload hf_hub -r {HF_REPO} -m {output_directory}"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Next steps\n",
+    "\n",
+    "- **Scale up**: Remove the `max_samples` caps and train on the full HH-RLHF dataset (~160k examples).\n",
+    "- **Try ORPO**: Use `config_orpo.yaml` for single-stage SFT + alignment without a reference model.\n",
+    "- **Iterate on beta**: Lower `beta` (e.g. 0.05) gives more aggressive alignment but risks reward hacking.\n",
+    "- **Evaluation**: Use [MT-Bench](https://github.com/lm-sys/FastChat/tree/main/fastchat/llm_judge) or\n",
+    "  [AlpacaEval](https://github.com/tatsu-lab/alpaca_eval) for rigorous alignment benchmarks.\n",
+    "- **GRPO**: For tasks with a programmatic reward (math, code), see the GRPO trainer docs."
+   ]
+  }
+ ],
+ "metadata": {
+  "accelerator": "GPU",
+  "colab": {
+   "gpuType": "A100",
+   "provenance": []
+  },
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "name": "python",
+   "version": "3.12.0"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

examples/alignment/config_dpo.yaml

@@ -0,0 +1,27 @@
+model_type: llm
+base_model: meta-llama/Llama-3.1-8B
+
+adapter:
+  type: lora
+  r: 16
+  alpha: 32
+  dropout: 0.05
+
+trainer:
+  type: dpo
+  epochs: 1
+  learning_rate: 5.0e-7
+  batch_size: 2
+  gradient_accumulation_steps: 8
+  beta: 0.1  # KL penalty coefficient — higher values keep the policy closer to the reference
+
+input_features:
+  - name: prompt
+    type: text
+
+output_features:
+  - name: chosen
+    type: text
+
+backend:
+  type: local

examples/alignment/config_grpo.yaml

@@ -0,0 +1,28 @@
+model_type: llm
+base_model: meta-llama/Llama-3.1-8B
+
+adapter:
+  type: lora
+  r: 16
+  alpha: 32
+
+trainer:
+  type: grpo
+  epochs: 1
+  learning_rate: 5.0e-7
+  batch_size: 1
+  gradient_accumulation_steps: 16
+  grpo_beta: 0.04        # KL penalty — keep the policy close to the reference model
+  grpo_epsilon: 0.2      # PPO clipping — bounds how much the policy can shift per step
+  grpo_num_generations: 4  # completions sampled per prompt; higher = better reward estimates
+
+input_features:
+  - name: prompt
+    type: text
+
+output_features:
+  - name: response
+    type: text
+
+backend:
+  type: local

examples/alignment/config_kto.yaml

@@ -0,0 +1,29 @@
+model_type: llm
+base_model: meta-llama/Llama-3.1-8B
+
+adapter:
+  type: lora
+  r: 16
+  alpha: 32
+  dropout: 0.05
+
+trainer:
+  type: kto
+  epochs: 1
+  learning_rate: 5.0e-7
+  batch_size: 2
+  gradient_accumulation_steps: 8
+  beta: 0.1
+  desirable_weight: 1.0    # weight for chosen (positive) samples
+  undesirable_weight: 1.0  # weight for rejected (negative) samples
+
+input_features:
+  - name: prompt
+    type: text
+
+output_features:
+  - name: response
+    type: text
+
+backend:
+  type: local

examples/alignment/config_orpo.yaml

@@ -0,0 +1,28 @@
+model_type: llm
+base_model: meta-llama/Llama-3.1-8B
+
+adapter:
+  type: lora
+  r: 16
+  alpha: 32
+  dropout: 0.05
+
+trainer:
+  type: orpo
+  epochs: 1
+  learning_rate: 5.0e-7
+  batch_size: 2
+  gradient_accumulation_steps: 8
+  # ORPO combines SFT and alignment in a single objective so no reference model
+  # forward passes are required and there is no beta KL penalty.
+
+input_features:
+  - name: prompt
+    type: text
+
+output_features:
+  - name: chosen
+    type: text
+
+backend:
+  type: local

examples/alignment/prepare_dataset.py

@@ -0,0 +1,132 @@
+"""Prepare the Anthropic HH-RLHF dataset for Ludwig alignment training.
+
+Downloads `Anthropic/hh-rlhf` from HuggingFace and converts it into the
+column format expected by Ludwig's DPO, KTO, and ORPO trainers.
+
+DPO / ORPO output:  train.csv, test.csv
+  Columns: prompt, chosen, rejected
+
+KTO output:         train_kto.csv, test_kto.csv
+  Columns: prompt, response, label  (label is True for chosen, False for rejected)
+
+The HH-RLHF dataset stores full multi-turn conversations as raw text with the
+pattern:
+    "\n\nHuman: <turn>\n\nAssistant: <turn>\n\nHuman: ...\n\nAssistant: ..."
+
+We extract the last Human turn as the prompt and the final Assistant turn as
+the response.  For DPO we do this for both `chosen` and `rejected` columns.
+"""
+
+import argparse
+import re
+
+import pandas as pd
+from datasets import load_dataset
+
+
+def extract_last_human_turn(conversation: str) -> str:
+    """Return the last Human turn from a raw HH-RLHF conversation string."""
+    human_turns = re.findall(r"\n\nHuman: (.*?)(?=\n\nAssistant:|\Z)", conversation, re.DOTALL)
+    if human_turns:
+        return human_turns[-1].strip()
+    return conversation.strip()
+
+
+def extract_last_assistant_turn(conversation: str) -> str:
+    """Return the last Assistant turn from a raw HH-RLHF conversation string."""
+    assistant_turns = re.findall(r"\n\nAssistant: (.*?)(?=\n\nHuman:|\Z)", conversation, re.DOTALL)
+    if assistant_turns:
+        return assistant_turns[-1].strip()
+    return ""
+
+
+def convert_split(split_data) -> tuple[pd.DataFrame, pd.DataFrame]:
+    """Convert one HF dataset split into DPO and KTO DataFrames."""
+    rows_dpo = []
+    rows_kto = []
+
+    for example in split_data:
+        chosen_conv = example["chosen"]
+        rejected_conv = example["rejected"]
+
+        prompt = extract_last_human_turn(chosen_conv)
+        chosen_response = extract_last_assistant_turn(chosen_conv)
+        rejected_response = extract_last_assistant_turn(rejected_conv)
+
+        if not prompt or not chosen_response or not rejected_response:
+            continue
+
+        rows_dpo.append(
+            {
+                "prompt": prompt,
+                "chosen": chosen_response,
+                "rejected": rejected_response,
+            }
+        )
+
+        # KTO: expand each pair into two rows with a boolean label
+        rows_kto.append({"prompt": prompt, "response": chosen_response, "label": True})
+        rows_kto.append({"prompt": prompt, "response": rejected_response, "label": False})
+
+    return pd.DataFrame(rows_dpo), pd.DataFrame(rows_kto)
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Prepare Anthropic HH-RLHF for Ludwig alignment training.")
+    parser.add_argument("--output_dir", default=".", help="Directory to write CSV files into.")
+    parser.add_argument(
+        "--max_train_samples",
+        type=int,
+        default=None,
+        help="Cap the number of training examples (useful for quick experiments).",
+    )
+    parser.add_argument(
+        "--max_test_samples",
+        type=int,
+        default=None,
+        help="Cap the number of test examples.",
+    )
+    args = parser.parse_args()
+
+    print("Downloading Anthropic/hh-rlhf …")
+    dataset = load_dataset("Anthropic/hh-rlhf")
+
+    train_split = dataset["train"]
+    test_split = dataset["test"]
+
+    if args.max_train_samples:
+        train_split = train_split.select(range(min(args.max_train_samples, len(train_split))))
+    if args.max_test_samples:
+        test_split = test_split.select(range(min(args.max_test_samples, len(test_split))))
+
+    print(f"Converting {len(train_split)} train examples …")
+    train_dpo, train_kto = convert_split(train_split)
+
+    print(f"Converting {len(test_split)} test examples …")
+    test_dpo, test_kto = convert_split(test_split)
+
+    import os
+
+    os.makedirs(args.output_dir, exist_ok=True)
+
+    train_path = os.path.join(args.output_dir, "train.csv")
+    test_path = os.path.join(args.output_dir, "test.csv")
+    train_kto_path = os.path.join(args.output_dir, "train_kto.csv")
+    test_kto_path = os.path.join(args.output_dir, "test_kto.csv")
+
+    train_dpo.to_csv(train_path, index=False)
+    test_dpo.to_csv(test_path, index=False)
+    train_kto.to_csv(train_kto_path, index=False)
+    test_kto.to_csv(test_kto_path, index=False)
+
+    print(f"\nDPO dataset:  {len(train_dpo)} train rows -> {train_path}")
+    print(f"              {len(test_dpo)} test rows  -> {test_path}")
+    print(f"KTO dataset:  {len(train_kto)} train rows -> {train_kto_path}")
+    print(f"              {len(test_kto)} test rows  -> {test_kto_path}")
+    print("\nColumns in DPO files:  prompt, chosen, rejected")
+    print("Columns in KTO files:  prompt, response, label")
+    print("\nDone.")
+
+
+if __name__ == "__main__":
+    main()

examples/alignment/train_dpo.py

@@ -0,0 +1,96 @@
+"""DPO alignment training with Ludwig.
+
+Usage:
+    python train_dpo.py --dataset train.csv
+    python train_dpo.py --dataset train.csv --epochs 3 --beta 0.05
+
+Prerequisites:
+    pip install "ludwig[llm]" datasets
+    export HUGGING_FACE_HUB_TOKEN="<your_token>"
+
+The dataset must have columns: prompt, chosen, rejected
+Use prepare_dataset.py to produce this file from Anthropic/hh-rlhf.
+"""
+
+import argparse
+import logging
+import os
+
+import yaml
+
+from ludwig.api import LudwigModel
+
+
+def build_config(epochs: int, learning_rate: float, beta: float, batch_size: int) -> dict:
+    raw = f"""
+model_type: llm
+base_model: meta-llama/Llama-3.1-8B
+
+adapter:
+  type: lora
+  r: 16
+  alpha: 32
+  dropout: 0.05
+
+trainer:
+  type: dpo
+  epochs: {epochs}
+  learning_rate: {learning_rate}
+  batch_size: {batch_size}
+  gradient_accumulation_steps: 8
+  beta: {beta}
+
+input_features:
+  - name: prompt
+    type: text
+
+output_features:
+  - name: chosen
+    type: text
+
+backend:
+  type: local
+"""
+    return yaml.safe_load(raw)
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Run DPO alignment training with Ludwig.")
+    parser.add_argument("--dataset", required=True, help="Path to the DPO CSV (prompt, chosen, rejected).")
+    parser.add_argument("--epochs", type=int, default=1)
+    parser.add_argument("--learning_rate", type=float, default=5e-7)
+    parser.add_argument("--beta", type=float, default=0.1, help="KL penalty coefficient.")
+    parser.add_argument("--batch_size", type=int, default=2)
+    parser.add_argument("--experiment_name", default="hh_rlhf_dpo")
+    parser.add_argument("--output_dir", default="results")
+    args = parser.parse_args()
+
+    token = os.environ.get("HUGGING_FACE_HUB_TOKEN") or os.environ.get("HF_TOKEN")
+    if not token:
+        raise OSError(
+            "Set HUGGING_FACE_HUB_TOKEN (or HF_TOKEN) before running. "
+            "You also need access approval for meta-llama/Llama-3.1-8B."
+        )
+
+    config = build_config(
+        epochs=args.epochs,
+        learning_rate=args.learning_rate,
+        beta=args.beta,
+        batch_size=args.batch_size,
+    )
+
+    model = LudwigModel(config=config, logging_level=logging.INFO)
+
+    train_stats, preprocessed_data, output_directory = model.train(
+        dataset=args.dataset,
+        experiment_name=args.experiment_name,
+        output_directory=args.output_dir,
+    )
+
+    print(f"\nTraining complete. Results saved to: {output_directory}")
+    print("To upload the model to HuggingFace Hub:")
+    print(f"    ludwig upload hf_hub -r <your_org>/<model_name> -m {output_directory}")
+
+
+if __name__ == "__main__":
+    main()

examples/alignment/train_grpo.py

@@ -0,0 +1,392 @@
+"""GRPO alignment training with Ludwig.
+
+Group Relative Policy Optimization (Shao et al., 2024 — DeepSeek-R1) trains a language
+model using a programmatic reward signal rather than preference pairs. Instead of
+"chosen vs rejected" data, you supply a reward function that scores each generated
+response. Ludwig samples grpo_num_generations completions per prompt, normalises
+rewards within the group, and applies a clipped PPO-style update.
+
+Usage (standalone):
+    python train_grpo.py
+    python train_grpo.py --epochs 2 --lr 3e-7 --output_dir my_run
+
+Usage (CLI with pre-scored dataset):
+    ludwig train --config config_grpo.yaml --dataset grpo_train.csv
+
+Prerequisites:
+    # Colab: !pip install "ludwig[llm]"
+    pip install "ludwig[llm]"
+    export HUGGING_FACE_HUB_TOKEN="<your_token>"
+    # You also need access approval for meta-llama/Llama-3.1-8B on HuggingFace Hub.
+
+How reward functions work with the Ludwig GRPO trainer
+-------------------------------------------------------
+The GRPO trainer expects a dataset where each row has:
+  - prompt   : the input question
+  - response : the correct / reference answer (used as a training target)
+
+The reward function is applied *before* training, in a data preparation step that
+scores each (prompt, response) pair and stores a "reward" column. During the GRPO
+update the trainer uses those scores — together with the group-normalised advantages
+computed across grpo_num_generations rollouts — to weight the policy gradient.
+
+NOTE: If a future Ludwig version adds a reward_fn parameter directly on LudwigModel
+or the GRPO trainer config, you can pass the callable there instead of pre-scoring.
+Check the Ludwig changelog for that API addition.
+"""
+
+import argparse
+import logging
+import os
+import re
+
+import pandas as pd
+import yaml
+
+from ludwig.api import LudwigModel
+
+logger = logging.getLogger(__name__)
+
+# ---------------------------------------------------------------------------
+# Dataset — 100 math word problems generated inline; no download needed
+# ---------------------------------------------------------------------------
+
+MATH_PROBLEMS = [
+    {
+        "prompt": "A baker has 48 cookies. She puts them equally into 6 bags. How many cookies are in each bag?",
+        "answer": "8",
+    },
+    {"prompt": "Tom has 15 apples. He gives 7 to his friend. How many apples does Tom have now?", "answer": "8"},
+    {"prompt": "A train travels 60 miles per hour. How far does it travel in 3 hours?", "answer": "180"},
+    {"prompt": "There are 24 students in a class. They form groups of 4. How many groups are there?", "answer": "6"},
+    {"prompt": "Maria saves $12 each week. How much does she save in 5 weeks?", "answer": "60"},
+    {"prompt": "A rectangle has a length of 8 cm and a width of 5 cm. What is its area?", "answer": "40"},
+    {
+        "prompt": "Jake has 3 boxes of crayons. Each box has 16 crayons. How many crayons does he have in total?",
+        "answer": "48",
+    },
+    {
+        "prompt": "A shop sells 35 items on Monday and 47 items on Tuesday. How many items were sold in total?",
+        "answer": "82",
+    },
+    {
+        "prompt": "There are 100 balloons. 38 are red and the rest are blue. How many blue balloons are there?",
+        "answer": "62",
+    },
+    {"prompt": "A car travels 90 km in 2 hours. What is its average speed in km/h?", "answer": "45"},
+    {"prompt": "Lucy reads 20 pages per day. How many pages does she read in 7 days?", "answer": "140"},
+    {"prompt": "A box holds 12 eggs. How many eggs are in 9 boxes?", "answer": "108"},
+    {"prompt": "A pool has 500 litres of water. 175 litres evaporate. How many litres remain?", "answer": "325"},
+    {"prompt": "There are 7 shelves with 9 books on each shelf. How many books are there in total?", "answer": "63"},
+    {"prompt": "Sam earns $8 per hour. How much does he earn in 6 hours?", "answer": "48"},
+    {"prompt": "A garden is 12 m long and 7 m wide. What is its perimeter?", "answer": "38"},
+    {"prompt": "There are 5 rows of chairs with 14 chairs in each row. How many chairs are there?", "answer": "70"},
+    {
+        "prompt": "Anna bakes 4 trays of muffins. Each tray has 12 muffins. How many muffins does she bake?",
+        "answer": "48",
+    },
+    {"prompt": "A rope is 72 cm long. It is cut into 8 equal pieces. How long is each piece?", "answer": "9"},
+    {"prompt": "Ben has 45 stickers. He gives 18 to his sister. How many stickers does Ben have left?", "answer": "27"},
+    {
+        "prompt": "A cinema sold 256 tickets on Saturday and 198 on Sunday. "
+        "How many tickets were sold over the weekend?",
+        "answer": "454",
+    },
+    {"prompt": "There are 360 minutes in 6 hours. How many minutes are in 1 hour?", "answer": "60"},
+    {"prompt": "A cyclist rides 15 km each day. How far does she ride in 4 days?", "answer": "60"},
+    {"prompt": "A bookstore has 5 shelves with 30 books each. How many books does it have in total?", "answer": "150"},
+    {"prompt": "A pizza is cut into 8 slices. If 3 slices are eaten, how many slices remain?", "answer": "5"},
+    {"prompt": "There are 4 quarters in a dollar. How many quarters are in $7?", "answer": "28"},
+    {"prompt": "A farmer has 120 eggs. He puts them in cartons of 12. How many cartons does he fill?", "answer": "10"},
+    {
+        "prompt": "A school has 480 pupils split equally across 6 classes. How many pupils are in each class?",
+        "answer": "80",
+    },
+    {
+        "prompt": "An ice cream shop sold 45 cones on Friday and 67 on Saturday. How many cones were sold in total?",
+        "answer": "112",
+    },
+    {"prompt": "A factory makes 250 units per day. How many units does it make in 5 days?", "answer": "1250"},
+    {
+        "prompt": "There are 18 players in a tournament. They are split into teams of 3. How many teams are there?",
+        "answer": "6",
+    },
+    {
+        "prompt": "A jar holds 96 sweets. If 4 children share them equally, how many sweets does each child get?",
+        "answer": "24",
+    },
+    {"prompt": "A plane flies 800 km in 2 hours. What is its average speed?", "answer": "400"},
+    {
+        "prompt": "A garden has 5 rows of flowers with 11 flowers in each row. How many flowers are there?",
+        "answer": "55",
+    },
+    {"prompt": "James has $200. He spends $74 on shoes. How much money does he have left?", "answer": "126"},
+    {"prompt": "A recipe uses 3 cups of flour per cake. How many cups are needed for 7 cakes?", "answer": "21"},
+    {"prompt": "There are 50 chairs in a hall. 13 are occupied. How many chairs are empty?", "answer": "37"},
+    {"prompt": "A clock ticks 60 times per minute. How many times does it tick in 5 minutes?", "answer": "300"},
+    {
+        "prompt": "A runner completes a 400 m lap in 80 seconds. How many laps does she run in 400 seconds?",
+        "answer": "5",
+    },
+    {"prompt": "There are 3 packs of pens with 12 pens each. How many pens are there in total?", "answer": "36"},
+    {"prompt": "A tank holds 200 gallons. It is currently 40% full. How many gallons are in the tank?", "answer": "80"},
+    {
+        "prompt": "A store has 84 items. They are arranged in 7 equal rows. How many items are in each row?",
+        "answer": "12",
+    },
+    {
+        "prompt": "There are 9 months until the concert. How many weeks is that (assuming 4 weeks per month)?",
+        "answer": "36",
+    },
+    {"prompt": "A frog jumps 3 m each jump. How far does it jump in 15 jumps?", "answer": "45"},
+    {"prompt": "A box of pencils costs $3. How much do 11 boxes cost?", "answer": "33"},
+    {"prompt": "There are 144 hours in 6 days. How many hours are in 1 day?", "answer": "24"},
+    {"prompt": "A pond has 300 fish. 75 are caught and released. How many fish remain?", "answer": "225"},
+    {"prompt": "A school bus seats 40 students. How many students can travel in 3 buses?", "answer": "120"},
+    {
+        "prompt": "There are 11 teams in a league. Each team plays every other team once. How many matches are there?",
+        "answer": "55",
+    },
+    {"prompt": "A bakery makes 60 loaves a day. How many loaves does it make in 2 weeks?", "answer": "840"},
+    {"prompt": "A square has sides of 9 cm. What is its perimeter?", "answer": "36"},
+    {"prompt": "An author writes 500 words per hour. How many words does she write in 3 hours?", "answer": "1500"},
+    {"prompt": "There are 72 hours in 3 days. How many hours are in 5 days?", "answer": "120"},
+    {"prompt": "A store discounts a $50 item by 20%. What is the sale price?", "answer": "40"},
+    {"prompt": "A farmer plants 8 seeds in each row. He has 9 rows. How many seeds does he plant?", "answer": "72"},
+    {"prompt": "A car park has 6 levels with 45 spaces each. How many spaces are there in total?", "answer": "270"},
+    {"prompt": "A marathon is 42 km. A runner has covered 28 km. How many km remain?", "answer": "14"},
+    {"prompt": "There are 30 days in a month. How many days are in 4 months?", "answer": "120"},
+    {"prompt": "A vending machine sells 15 drinks per hour. How many drinks does it sell in 8 hours?", "answer": "120"},
+    {"prompt": "There are 6 strings on a guitar. How many strings are on 9 guitars?", "answer": "54"},
+    {"prompt": "A worker earns $15 per hour and works 8 hours. How much does she earn?", "answer": "120"},
+    {"prompt": "A square room has an area of 64 m². What is the length of each side?", "answer": "8"},
+    {"prompt": "A jar has 5 red marbles and 8 blue marbles. How many marbles are there in total?", "answer": "13"},
+    {"prompt": "A truck carries 2 tonnes per trip. How many tonnes does it carry in 7 trips?", "answer": "14"},
+    {"prompt": "There are 32 students. Half are girls. How many girls are there?", "answer": "16"},
+    {"prompt": "A patio is 6 m wide and 9 m long. What is its area?", "answer": "54"},
+    {"prompt": "A swimmer does 50 laps per session. How many laps does she do in 6 sessions?", "answer": "300"},
+    {"prompt": "A bag of rice weighs 5 kg. How much do 8 bags weigh?", "answer": "40"},
+    {"prompt": "There are 7 days in a week. How many days are in 13 weeks?", "answer": "91"},
+    {"prompt": "A factory produces 1200 items in 4 hours. How many items per hour?", "answer": "300"},
+    {
+        "prompt": "A road is 3.5 km long. Two cars start at each end and drive toward each other "
+        "at 1.75 km/h each. How long until they meet (in hours)?",
+        "answer": "1",
+    },
+    {"prompt": "There are 18 biscuits. Each person eats 3. How many people can eat?", "answer": "6"},
+    {
+        "prompt": "A pool requires 8 hours to fill. After 5 hours, how much is left to fill (as a fraction)?",
+        "answer": "3/8",
+    },
+    {"prompt": "A box has 4 layers with 25 chocolates per layer. How many chocolates are in the box?", "answer": "100"},
+    {"prompt": "A phone battery lasts 12 hours. After 9 hours of use, what percentage remains?", "answer": "25"},
+    {"prompt": "There are 40 red and 25 blue tiles. How many tiles are there in total?", "answer": "65"},
+    {"prompt": "A rope is 54 m long. It is divided into 9 equal pieces. How long is each piece?", "answer": "6"},
+    {"prompt": "A bag has 3 green, 4 yellow, and 5 purple balls. How many balls are there?", "answer": "12"},
+    {"prompt": "Each page has 30 lines. How many lines are on 8 pages?", "answer": "240"},
+    {"prompt": "A worker packs 6 boxes per hour. How many boxes in 9 hours?", "answer": "54"},
+    {"prompt": "There are 1000 metres in a kilometre. How many metres in 7.5 km?", "answer": "7500"},
+    {"prompt": "A container holds 5 litres. How many containers are needed for 35 litres?", "answer": "7"},
+    {
+        "prompt": "A car travels 110 km in 2 hours. How far does it travel in 5 hours at the same speed?",
+        "answer": "275",
+    },
+    {"prompt": "There are 52 cards in a deck. How many cards are in 3 decks?", "answer": "156"},
+    {"prompt": "A hotel has 12 floors with 18 rooms each. How many rooms are there?", "answer": "216"},
+    {
+        "prompt": "A pie is split into 6 equal slices. Two people each eat 2 slices. How many slices remain?",
+        "answer": "2",
+    },
+    {"prompt": "A printing press produces 40 pages per minute. How many pages in 15 minutes?", "answer": "600"},
+    {"prompt": "There are 200 pupils. 45% are boys. How many boys are there?", "answer": "90"},
+    {"prompt": "A wall is 3 m tall and 8 m wide. What is its area?", "answer": "24"},
+    {"prompt": "A car uses 6 litres of fuel per 100 km. How much fuel is needed for 300 km?", "answer": "18"},
+    {"prompt": "A team scores 3 goals in each of 6 matches. How many goals in total?", "answer": "18"},
+    {"prompt": "There are 11 rows of seats with 20 seats each. How many seats are there?", "answer": "220"},
+    {"prompt": "A box contains 50 nails. After using 23, how many remain?", "answer": "27"},
+    {"prompt": "A wheel turns 360 degrees in one full rotation. How many degrees in 5 rotations?", "answer": "1800"},
+    {"prompt": "A shop receives 3 deliveries of 40 items each. How many items did it receive?", "answer": "120"},
+    {"prompt": "There are 15 biscuits on a plate. 6 are eaten. How many remain?", "answer": "9"},
+]
+
+
+# ---------------------------------------------------------------------------
+# Reward function
+# ---------------------------------------------------------------------------
+
+
+def reward_fn(prompt: str, response: str) -> float:
+    """Score a model response by checking if it contains the correct numerical answer.
+
+    Returns 1.0 if the response contains the expected answer, 0.0 otherwise. The correct answer is looked up from the
+    global ANSWER_LOOKUP dict which is built from MATH_PROBLEMS at startup.
+
+    This is a simple exact-match reward. Real applications might use an LLM judge, a code execution sandbox, or a
+    symbolic verifier.
+    """
+    expected = ANSWER_LOOKUP.get(prompt.strip())
+    if expected is None:
+        return 0.0
+    # Accept the answer if it appears as a standalone number/fraction in the response
+    pattern = r"(?<![.\d])" + re.escape(expected) + r"(?![.\d])"
+    return 1.0 if re.search(pattern, response) else 0.0
+
+
+# Map from prompt text → expected answer string, built once at module load time.
+ANSWER_LOOKUP: dict[str, str] = {p["prompt"]: p["answer"] for p in MATH_PROBLEMS}
+
+
+# ---------------------------------------------------------------------------
+# Dataset preparation — apply reward_fn to produce a scored DataFrame
+# ---------------------------------------------------------------------------
+
+
+def build_dataset(reward_function) -> pd.DataFrame:
+    """Build a training DataFrame by applying the reward function to each example.
+
+    For GRPO the dataset needs at minimum:
+      - prompt   : the input text
+      - response : the reference / target text
+      - reward   : a float score for each (prompt, response) pair
+
+    NOTE: The Ludwig GRPO trainer does not yet accept a reward_fn callable at
+    train-time (as of v0.11.dev). Instead, rewards are pre-computed here and
+    stored in a 'reward' column that the trainer reads from the dataset. If a
+    future version exposes a reward_fn parameter on LudwigModel or the GRPO
+    config, you can pass `reward_fn=reward_function` there directly.
+    """
+    rows = []
+    for item in MATH_PROBLEMS:
+        prompt = item["prompt"]
+        response = item["answer"]
+        reward = reward_function(prompt, response)
+        rows.append({"prompt": prompt, "response": response, "reward": reward})
+    df = pd.DataFrame(rows)
+    logger.info(
+        "Dataset built: %d examples, mean reward=%.3f",
+        len(df),
+        df["reward"].mean(),
+    )
+    return df
+
+
+# ---------------------------------------------------------------------------
+# Config builder
+# ---------------------------------------------------------------------------
+
+
+def build_config(epochs: int, learning_rate: float, batch_size: int) -> dict:
+    raw = f"""
+model_type: llm
+base_model: meta-llama/Llama-3.1-8B
+
+adapter:
+  type: lora
+  r: 16
+  alpha: 32
+
+trainer:
+  type: grpo
+  epochs: {epochs}
+  learning_rate: {learning_rate}
+  batch_size: {batch_size}
+  gradient_accumulation_steps: 16
+  grpo_beta: 0.04
+  grpo_epsilon: 0.2
+  grpo_num_generations: 4
+
+input_features:
+  - name: prompt
+    type: text
+
+output_features:
+  - name: response
+    type: text
+
+backend:
+  type: local
+"""
+    return yaml.safe_load(raw)
+
+
+# ---------------------------------------------------------------------------
+# GPU check
+# ---------------------------------------------------------------------------
+
+
+def check_gpu():
+    try:
+        import torch
+
+        if torch.cuda.is_available() and torch.cuda.device_count() > 0:
+            name = torch.cuda.get_device_name(0)
+            vram = torch.cuda.get_device_properties(0).total_memory / 1024**3
+            logger.info("GPU detected: %s (%.1f GiB VRAM)", name, vram)
+            if vram < 20:
+                logger.warning(
+                    "Only %.1f GiB VRAM detected. Llama-3.1-8B requires at least 40 GiB "
+                    "for GRPO training. Consider using a smaller base model or enabling "
+                    "quantisation (e.g. bitsandbytes 4-bit).",
+                    vram,
+                )
+        else:
+            logger.warning(
+                "No GPU detected. GRPO training on a 7-8B model will be extremely slow on CPU. "
+                "On Colab, go to Runtime > Change runtime type and select a GPU."
+            )
+    except ImportError:
+        pass
+
+
+# ---------------------------------------------------------------------------
+# Main
+# ---------------------------------------------------------------------------
+
+
+def main():
+    logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s")
+
+    parser = argparse.ArgumentParser(description="GRPO alignment training with Ludwig.")
+    parser.add_argument("--epochs", type=int, default=1)
+    parser.add_argument("--lr", type=float, default=5e-7)
+    parser.add_argument("--batch_size", type=int, default=1)
+    parser.add_argument("--experiment_name", default="math_grpo")
+    parser.add_argument("--output_dir", default="results")
+    args = parser.parse_args()
+
+    # --- HuggingFace token ---
+    token = os.environ.get("HUGGING_FACE_HUB_TOKEN") or os.environ.get("HF_TOKEN")
+    if not token:
+        raise OSError(
+            "Set HUGGING_FACE_HUB_TOKEN (or HF_TOKEN) before running. "
+            "You also need access approval for meta-llama/Llama-3.1-8B on HuggingFace Hub."
+        )
+
+    check_gpu()
+
+    # --- Build dataset with pre-computed rewards ---
+    df = build_dataset(reward_fn)
+
+    # --- Build Ludwig config ---
+    config = build_config(
+        epochs=args.epochs,
+        learning_rate=args.lr,
+        batch_size=args.batch_size,
+    )
+
+    # --- Train ---
+    model = LudwigModel(config=config, logging_level=logging.INFO)
+
+    train_stats, _, output_directory = model.train(
+        dataset=df,
+        experiment_name=args.experiment_name,
+        output_directory=args.output_dir,
+    )
+
+    print(f"\nTraining complete. Results saved to: {output_directory}")
+    print("To upload the aligned model to HuggingFace Hub:")
+    print(f"    ludwig upload hf_hub -r <your_org>/<model_name> -m {output_directory}")
+
+
+if __name__ == "__main__":
+    main()

examples/anomaly_detection/README.md

@@ -0,0 +1,119 @@
+# Anomaly Detection with Deep SVDD, SAD, and DROCC
+
+[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ludwig-ai/ludwig/blob/main/examples/anomaly_detection/anomaly_detection.ipynb)
+
+This example shows how to train an anomaly detection model with Ludwig using the `anomaly` output feature type. The model learns a compact representation of "normal" sensor data using three complementary hypersphere-based objectives:
+
+- **Deep SVDD** — unsupervised, trains only on normal samples
+- **Deep SAD** — semi-supervised, uses a small set of labeled anomalies at training time
+- **DROCC** — unsupervised with adversarial robustness, recommended for expressive encoders
+
+At inference time each sample receives an `anomaly_score` equal to its squared distance from the learned hypersphere centre. Higher scores indicate more anomalous samples.
+
+## Prerequisites
+
+```bash
+pip install ludwig
+```
+
+## Dataset
+
+The example uses a synthetic sensor dataset with four numeric features (`sensor_a`, `sensor_b`, `sensor_c`, `timestamp_hour`). Normal samples are drawn from a Gaussian distribution centred at the origin; anomalous samples have a large offset. The train split contains **only normal samples**; the test split contains both normal and anomalous samples for evaluation.
+
+## Loss variants
+
+### Deep SVDD (unsupervised)
+
+```yaml
+output_features:
+  - name: anomaly
+    type: anomaly
+    loss:
+      type: deep_svdd
+      nu: 0.1   # fraction of points allowed outside the hypersphere
+```
+
+Hard-boundary objective: minimise the mean squared distance of all normal training representations to the hypersphere centre `c`. The `nu` parameter controls soft-boundary relaxation (set to `0` for hard SVDD).
+
+Full config: [`config_deep_svdd.yaml`](config_deep_svdd.yaml)
+
+### Deep SAD (semi-supervised)
+
+```yaml
+output_features:
+  - name: anomaly
+    type: anomaly
+    loss:
+      type: deep_sad
+      eta: 1.0   # weight for the labeled anomaly repulsion term
+```
+
+Extends Deep SVDD with labeled anomaly support. Normal and unlabeled samples (label `0` or `-1`) are pulled toward `c`; labeled anomalies (label `1`) are pushed away. Provide a small fraction of labeled anomaly rows in the training data with `anomaly=1`.
+
+Full config: [`config_deep_sad.yaml`](config_deep_sad.yaml)
+
+### DROCC (robust unsupervised)
+
+```yaml
+output_features:
+  - name: anomaly
+    type: anomaly
+    loss:
+      type: drocc
+      perturbation_strength: 0.1
+      num_perturbation_steps: 5
+```
+
+Prevents hypersphere collapse via an adversarial perturbation regulariser. Recommended when using expressive encoders (e.g. transformers) that are prone to degenerate solutions where all representations collapse to a single point.
+
+Full config: [`config_drocc.yaml`](config_drocc.yaml)
+
+## Running the example
+
+### CLI
+
+```bash
+# Train
+ludwig train --config config_deep_svdd.yaml --dataset /tmp/sensors_train.csv
+
+# Predict (score test samples)
+ludwig predict --model_path results/experiment_run/model \
+               --dataset /tmp/sensors_test.csv
+
+# Evaluate (requires labeled anomaly column in test CSV)
+ludwig evaluate --model_path results/experiment_run/model \
+                --dataset /tmp/sensors_test.csv
+```
+
+### Python API
+
+```python
+import pandas as pd
+from ludwig.api import LudwigModel
+
+# Load data
+train_df = pd.read_csv("/tmp/sensors_train.csv")
+test_df = pd.read_csv("/tmp/sensors_test.csv")
+
+# Train
+model = LudwigModel("config_deep_svdd.yaml", logging_level="ERROR")
+results = model.train(dataset=train_df)
+
+# Predict — returns a DataFrame with anomaly_score_predictions column
+predictions, _ = model.predict(dataset=test_df)
+print(predictions[["anomaly_anomaly_score_predictions"]].describe())
+```
+
+For a full walkthrough including score distribution plots and AUC comparison, open the notebook in Colab:
+
+[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ludwig-ai/ludwig/blob/main/examples/anomaly_detection/anomaly_detection.ipynb)
+
+## Files
+
+| File                      | Description                               |
+| ------------------------- | ----------------------------------------- |
+| `anomaly_detection.ipynb` | End-to-end Colab notebook                 |
+| `config_deep_svdd.yaml`   | Deep SVDD config                          |
+| `config_deep_sad.yaml`    | Deep SAD (semi-supervised) config         |
+| `config_drocc.yaml`       | DROCC config                              |
+| `train.py`                | Standalone training and evaluation script |

examples/anomaly_detection/anomaly_detection.ipynb

@@ -0,0 +1,486 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "7fb27b941602401d91542211134fc71a",
+   "metadata": {},
+   "source": [
+    "# Anomaly Detection with Deep SVDD, SAD, and DROCC\n",
+    "\n",
+    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ludwig-ai/ludwig/blob/main/examples/anomaly_detection/anomaly_detection.ipynb)\n",
+    "\n",
+    "This notebook demonstrates **unsupervised anomaly detection** using Ludwig's `anomaly` output feature.\n",
+    "The model learns a compact representation of normal data inside a hypersphere in latent space.\n",
+    "At inference time, each sample is scored by its squared distance to the centre of that hypersphere:\n",
+    "samples far from the centre are flagged as anomalies.\n",
+    "\n",
+    "```\n",
+    "anomaly_score = ||z - c||^2\n",
+    "```\n",
+    "\n",
+    "**What this notebook covers:**\n",
+    "\n",
+    "1. Generating a synthetic sensor dataset (normal vs anomalous readings)\n",
+    "2. Training with **Deep SVDD** (fully unsupervised)\n",
+    "3. Visualising the anomaly score distribution for normal vs anomalous samples\n",
+    "4. Training with **Deep SAD** (semi-supervised — uses a few labeled anomalies)\n",
+    "5. Training with **DROCC** (robust unsupervised — adversarial perturbations prevent collapse)\n",
+    "6. Comparing AUC-ROC across all three methods"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "acae54e37e7d407bbb7b55eff062a284",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!pip install ludwig --quiet"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "9a63283cbaf04dbcab1f6479b197f3a8",
+   "metadata": {},
+   "source": [
+    "## Generate synthetic sensor data"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8dd0d8092fe74a7c96281538738b07e2",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "import pandas as pd\n",
+    "\n",
+    "RNG = np.random.default_rng(42)\n",
+    "\n",
+    "N_NORMAL = 800\n",
+    "N_ANOMALY = 200\n",
+    "\n",
+    "# Normal samples: Gaussian near origin\n",
+    "normal_df = pd.DataFrame(\n",
+    "    {\n",
+    "        \"sensor_a\": RNG.normal(0.0, 1.0, N_NORMAL),\n",
+    "        \"sensor_b\": RNG.normal(0.0, 1.0, N_NORMAL),\n",
+    "        \"sensor_c\": RNG.normal(0.0, 1.0, N_NORMAL),\n",
+    "        \"timestamp_hour\": RNG.integers(0, 24, N_NORMAL).astype(float),\n",
+    "        \"anomaly\": 0.0,\n",
+    "    }\n",
+    ")\n",
+    "\n",
+    "# Anomalous samples: large offset from origin\n",
+    "anomaly_df = pd.DataFrame(\n",
+    "    {\n",
+    "        \"sensor_a\": RNG.normal(6.0, 1.0, N_ANOMALY),\n",
+    "        \"sensor_b\": RNG.normal(6.0, 1.0, N_ANOMALY),\n",
+    "        \"sensor_c\": RNG.normal(6.0, 1.0, N_ANOMALY),\n",
+    "        \"timestamp_hour\": RNG.integers(0, 24, N_ANOMALY).astype(float),\n",
+    "        \"anomaly\": 1.0,\n",
+    "    }\n",
+    ")\n",
+    "\n",
+    "# ---------------------------------------------------------------\n",
+    "# Train split: ONLY normal samples (anomaly detection is unsupervised).\n",
+    "# Validation / test splits: mix of normal and anomalous.\n",
+    "# split column: 0=train, 1=validation, 2=test\n",
+    "# ---------------------------------------------------------------\n",
+    "\n",
+    "normal_idx = normal_df.index.tolist()\n",
+    "RNG.shuffle(normal_idx)\n",
+    "n_train = int(0.7 * len(normal_idx))\n",
+    "n_val = int(0.15 * len(normal_idx))\n",
+    "\n",
+    "normal_df[\"split\"] = 2  # test by default\n",
+    "normal_df.loc[normal_idx[:n_train], \"split\"] = 0\n",
+    "normal_df.loc[normal_idx[n_train : n_train + n_val], \"split\"] = 1\n",
+    "\n",
+    "anom_idx = anomaly_df.index.tolist()\n",
+    "RNG.shuffle(anom_idx)\n",
+    "n_val_anom = len(anom_idx) // 2\n",
+    "anomaly_df[\"split\"] = 2\n",
+    "anomaly_df.loc[anom_idx[:n_val_anom], \"split\"] = 1\n",
+    "\n",
+    "# Training CSV: only normal rows\n",
+    "train_df = normal_df[normal_df[\"split\"] == 0].copy()\n",
+    "\n",
+    "# Test CSV: validation + test rows (normal and anomalous), used for scoring\n",
+    "val_test_df = pd.concat([normal_df[normal_df[\"split\"] != 0], anomaly_df], ignore_index=True)\n",
+    "\n",
+    "train_df.to_csv(\"/tmp/sensors_train.csv\", index=False)\n",
+    "val_test_df.to_csv(\"/tmp/sensors_test.csv\", index=False)\n",
+    "\n",
+    "print(f\"Train samples : {len(train_df)} (all normal)\")\n",
+    "print(f\"Test  samples : {len(val_test_df)}\")\n",
+    "print(f\"  Normal    : {(val_test_df['anomaly'] == 0).sum()}\")\n",
+    "print(f\"  Anomalous : {(val_test_df['anomaly'] == 1).sum()}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "72eea5119410473aa328ad9291626812",
+   "metadata": {},
+   "source": [
+    "## Train: Deep SVDD\n",
+    "\n",
+    "Deep SVDD (Ruff et al., ICML 2018) is the simplest variant: it minimises the mean squared\n",
+    "distance of all training representations to a hypersphere centre `c`, learned during training.\n",
+    "Only normal samples are used for training — no anomaly labels are required.\n",
+    "\n",
+    "The `nu` parameter controls the soft-boundary relaxation: `nu=0.1` allows up to 10% of\n",
+    "training points outside the hypersphere."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8edb47106e1a46a883d545849b8ab81b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import yaml\n",
+    "\n",
+    "from ludwig.api import LudwigModel\n",
+    "\n",
+    "config_svdd_str = \"\"\"\n",
+    "model_type: ecd\n",
+    "\n",
+    "input_features:\n",
+    "  - name: sensor_a\n",
+    "    type: number\n",
+    "    preprocessing:\n",
+    "      normalization: zscore\n",
+    "  - name: sensor_b\n",
+    "    type: number\n",
+    "    preprocessing:\n",
+    "      normalization: zscore\n",
+    "  - name: sensor_c\n",
+    "    type: number\n",
+    "    preprocessing:\n",
+    "      normalization: zscore\n",
+    "  - name: timestamp_hour\n",
+    "    type: number\n",
+    "    preprocessing:\n",
+    "      normalization: zscore\n",
+    "\n",
+    "output_features:\n",
+    "  - name: anomaly\n",
+    "    type: anomaly\n",
+    "    loss:\n",
+    "      type: deep_svdd\n",
+    "      nu: 0.1\n",
+    "\n",
+    "trainer:\n",
+    "  epochs: 20\n",
+    "  learning_rate: 0.001\n",
+    "\n",
+    "combiner:\n",
+    "  type: concat\n",
+    "  fc_layers:\n",
+    "    - output_size: 64\n",
+    "    - output_size: 32\n",
+    "\"\"\"\n",
+    "\n",
+    "config_svdd = yaml.safe_load(config_svdd_str)\n",
+    "\n",
+    "model_svdd = LudwigModel(config_svdd, logging_level=30)\n",
+    "train_stats_svdd, _, _ = model_svdd.train(dataset=train_df)\n",
+    "\n",
+    "print(\"Deep SVDD training complete.\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "10185d26023b46108eb7d9f57d49d2b3",
+   "metadata": {},
+   "source": [
+    "## Evaluate: score distribution\n",
+    "\n",
+    "Run inference on the test set (which contains both normal and anomalous samples) and plot\n",
+    "the distribution of `anomaly_score_predictions`. A good model will produce clearly separated\n",
+    "distributions for the two classes."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8763a12b2bbd4a93a75aff182afb95dc",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import matplotlib.pyplot as plt\n",
+    "from sklearn.metrics import roc_auc_score\n",
+    "\n",
+    "preds_svdd, _ = model_svdd.predict(dataset=val_test_df)\n",
+    "\n",
+    "score_col = \"anomaly_anomaly_score_predictions\"\n",
+    "scores_svdd = preds_svdd[score_col].values\n",
+    "true_labels = val_test_df[\"anomaly\"].values\n",
+    "\n",
+    "auc_svdd = roc_auc_score(true_labels, scores_svdd)\n",
+    "print(f\"Deep SVDD  AUC-ROC: {auc_svdd:.4f}\")\n",
+    "\n",
+    "# --- Plot score distribution ---\n",
+    "fig, ax = plt.subplots(figsize=(8, 4))\n",
+    "\n",
+    "ax.hist(scores_svdd[true_labels == 0], bins=40, alpha=0.6, label=\"Normal\", color=\"steelblue\")\n",
+    "ax.hist(scores_svdd[true_labels == 1], bins=40, alpha=0.6, label=\"Anomalous\", color=\"tomato\")\n",
+    "\n",
+    "ax.set_xlabel(\"Anomaly score  ||z - c||^2\")\n",
+    "ax.set_ylabel(\"Count\")\n",
+    "ax.set_title(f\"Deep SVDD — anomaly score distribution  (AUC = {auc_svdd:.3f})\")\n",
+    "ax.legend()\n",
+    "plt.tight_layout()\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7623eae2785240b9bd12b16a66d81610",
+   "metadata": {},
+   "source": [
+    "## Try Deep SAD (semi-supervised)\n",
+    "\n",
+    "Deep SAD (Ruff et al., ICLR 2020) extends Deep SVDD by incorporating a small set of\n",
+    "**labeled anomalies** during training. Normal/unlabeled samples are pulled toward centre `c`;\n",
+    "labeled anomalies are pushed away. This typically improves score separation when even a\n",
+    "handful of anomaly examples are available.\n",
+    "\n",
+    "In the dataset, set `anomaly=1` for labeled anomaly rows, `anomaly=0` for normal rows, and\n",
+    "`anomaly=-1` (or leave the column absent) for unlabeled rows that should be ignored by the\n",
+    "repulsion term.\n",
+    "\n",
+    "Here we inject ~10% labeled anomalies into the training split (`labeled_anomalies_ratio=0.1`)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "7cdc8c89c7104fffa095e18ddfef8986",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "config_sad_str = \"\"\"\n",
+    "model_type: ecd\n",
+    "\n",
+    "input_features:\n",
+    "  - name: sensor_a\n",
+    "    type: number\n",
+    "    preprocessing:\n",
+    "      normalization: zscore\n",
+    "  - name: sensor_b\n",
+    "    type: number\n",
+    "    preprocessing:\n",
+    "      normalization: zscore\n",
+    "  - name: sensor_c\n",
+    "    type: number\n",
+    "    preprocessing:\n",
+    "      normalization: zscore\n",
+    "  - name: timestamp_hour\n",
+    "    type: number\n",
+    "    preprocessing:\n",
+    "      normalization: zscore\n",
+    "\n",
+    "output_features:\n",
+    "  - name: anomaly\n",
+    "    type: anomaly\n",
+    "    loss:\n",
+    "      type: deep_sad\n",
+    "      eta: 1.0\n",
+    "\n",
+    "trainer:\n",
+    "  epochs: 20\n",
+    "  learning_rate: 0.001\n",
+    "\n",
+    "combiner:\n",
+    "  type: concat\n",
+    "  fc_layers:\n",
+    "    - output_size: 64\n",
+    "    - output_size: 32\n",
+    "\"\"\"\n",
+    "\n",
+    "config_sad = yaml.safe_load(config_sad_str)\n",
+    "\n",
+    "# Inject ~10% labeled anomalies into the training set\n",
+    "N_LABELED = max(1, int(0.1 * len(train_df)))\n",
+    "labeled_anom = anomaly_df.sample(n=N_LABELED, random_state=0).copy()\n",
+    "labeled_anom[\"split\"] = 0\n",
+    "sad_train_df = pd.concat([train_df, labeled_anom], ignore_index=True)\n",
+    "\n",
+    "print(\n",
+    "    f\"Deep SAD training set: {len(sad_train_df)} rows  \"\n",
+    "    f\"({N_LABELED} labeled anomalies = \"\n",
+    "    f\"{100 * N_LABELED / len(sad_train_df):.1f}%)\"\n",
+    ")\n",
+    "\n",
+    "model_sad = LudwigModel(config_sad, logging_level=30)\n",
+    "train_stats_sad, _, _ = model_sad.train(dataset=sad_train_df)\n",
+    "\n",
+    "preds_sad, _ = model_sad.predict(dataset=val_test_df)\n",
+    "scores_sad = preds_sad[score_col].values\n",
+    "auc_sad = roc_auc_score(true_labels, scores_sad)\n",
+    "print(f\"Deep SAD   AUC-ROC: {auc_sad:.4f}\")\n",
+    "\n",
+    "# --- Plot ---\n",
+    "fig, ax = plt.subplots(figsize=(8, 4))\n",
+    "ax.hist(scores_sad[true_labels == 0], bins=40, alpha=0.6, label=\"Normal\", color=\"steelblue\")\n",
+    "ax.hist(scores_sad[true_labels == 1], bins=40, alpha=0.6, label=\"Anomalous\", color=\"tomato\")\n",
+    "ax.set_xlabel(\"Anomaly score  ||z - c||^2\")\n",
+    "ax.set_ylabel(\"Count\")\n",
+    "ax.set_title(f\"Deep SAD — score distribution  (AUC = {auc_sad:.3f})\")\n",
+    "ax.legend()\n",
+    "plt.tight_layout()\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b118ea5561624da68c537baed56e602f",
+   "metadata": {},
+   "source": [
+    "## Try DROCC\n",
+    "\n",
+    "DROCC (Goyal et al., ICML 2020) prevents hypersphere **collapse** — a failure mode where an\n",
+    "expressive encoder maps all inputs to the same point, achieving a trivially low loss while\n",
+    "learning nothing useful. It does so by generating adversarial perturbations around each\n",
+    "normal training point and penalising the model if the perturbed points score as normal.\n",
+    "\n",
+    "Key parameters:\n",
+    "- `perturbation_strength`: magnitude of adversarial perturbations (default `0.1`)\n",
+    "- `num_perturbation_steps`: gradient ascent steps per sample (default `5`)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "938c804e27f84196a10c8828c723f798",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "config_drocc_str = \"\"\"\n",
+    "model_type: ecd\n",
+    "\n",
+    "input_features:\n",
+    "  - name: sensor_a\n",
+    "    type: number\n",
+    "    preprocessing:\n",
+    "      normalization: zscore\n",
+    "  - name: sensor_b\n",
+    "    type: number\n",
+    "    preprocessing:\n",
+    "      normalization: zscore\n",
+    "  - name: sensor_c\n",
+    "    type: number\n",
+    "    preprocessing:\n",
+    "      normalization: zscore\n",
+    "  - name: timestamp_hour\n",
+    "    type: number\n",
+    "    preprocessing:\n",
+    "      normalization: zscore\n",
+    "\n",
+    "output_features:\n",
+    "  - name: anomaly\n",
+    "    type: anomaly\n",
+    "    loss:\n",
+    "      type: drocc\n",
+    "      perturbation_strength: 0.1\n",
+    "      num_perturbation_steps: 5\n",
+    "\n",
+    "trainer:\n",
+    "  epochs: 20\n",
+    "  learning_rate: 0.001\n",
+    "\n",
+    "combiner:\n",
+    "  type: concat\n",
+    "  fc_layers:\n",
+    "    - output_size: 64\n",
+    "    - output_size: 32\n",
+    "\"\"\"\n",
+    "\n",
+    "config_drocc = yaml.safe_load(config_drocc_str)\n",
+    "\n",
+    "model_drocc = LudwigModel(config_drocc, logging_level=30)\n",
+    "train_stats_drocc, _, _ = model_drocc.train(dataset=train_df)\n",
+    "\n",
+    "preds_drocc, _ = model_drocc.predict(dataset=val_test_df)\n",
+    "scores_drocc = preds_drocc[score_col].values\n",
+    "auc_drocc = roc_auc_score(true_labels, scores_drocc)\n",
+    "print(f\"DROCC      AUC-ROC: {auc_drocc:.4f}\")\n",
+    "\n",
+    "# --- Plot ---\n",
+    "fig, ax = plt.subplots(figsize=(8, 4))\n",
+    "ax.hist(scores_drocc[true_labels == 0], bins=40, alpha=0.6, label=\"Normal\", color=\"steelblue\")\n",
+    "ax.hist(scores_drocc[true_labels == 1], bins=40, alpha=0.6, label=\"Anomalous\", color=\"tomato\")\n",
+    "ax.set_xlabel(\"Anomaly score  ||z - c||^2\")\n",
+    "ax.set_ylabel(\"Count\")\n",
+    "ax.set_title(f\"DROCC — score distribution  (AUC = {auc_drocc:.3f})\")\n",
+    "ax.legend()\n",
+    "plt.tight_layout()\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "504fb2a444614c0babb325280ed9130a",
+   "metadata": {},
+   "source": [
+    "## Summary\n",
+    "\n",
+    "The table below compares the three loss variants on the synthetic sensor dataset.\n",
+    "AUC-ROC close to 1.0 means the model can almost perfectly rank anomalous samples above\n",
+    "normal ones. The separation ratio is the mean anomaly score divided by the mean normal score:\n",
+    "a ratio much greater than 1 indicates clear separation in score space."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "59bbdb311c014d738909a11f9e486628",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "summary_rows = []\n",
+    "for name, scores, auc in [\n",
+    "    (\"Deep SVDD (unsupervised)\", scores_svdd, auc_svdd),\n",
+    "    (\"Deep SAD  (semi-supervised)\", scores_sad, auc_sad),\n",
+    "    (\"DROCC     (robust unsup.)\", scores_drocc, auc_drocc),\n",
+    "]:\n",
+    "    normal_s = scores[true_labels == 0]\n",
+    "    anom_s = scores[true_labels == 1]\n",
+    "    sep = anom_s.mean() / (normal_s.mean() + 1e-9)\n",
+    "    summary_rows.append(\n",
+    "        {\n",
+    "            \"Method\": name,\n",
+    "            \"AUC-ROC\": round(auc, 4),\n",
+    "            \"Mean normal score\": round(float(normal_s.mean()), 4),\n",
+    "            \"Mean anomaly score\": round(float(anom_s.mean()), 4),\n",
+    "            \"Separation ratio\": round(float(sep), 2),\n",
+    "        }\n",
+    "    )\n",
+    "\n",
+    "summary_df = pd.DataFrame(summary_rows)\n",
+    "print(summary_df.to_string(index=False))"
+   ]
+  }
+ ],
+ "metadata": {
+  "accelerator": "GPU",
+  "colab": {
+   "provenance": []
+  },
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "name": "python",
+   "version": "3.12.0"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

examples/anomaly_detection/config_deep_sad.yaml

@@ -0,0 +1,44 @@
+model_type: ecd
+
+input_features:
+  - name: sensor_a
+    type: number
+    preprocessing:
+      normalization: zscore
+  - name: sensor_b
+    type: number
+    preprocessing:
+      normalization: zscore
+  - name: sensor_c
+    type: number
+    preprocessing:
+      normalization: zscore
+  - name: timestamp_hour
+    type: number
+    preprocessing:
+      normalization: zscore
+
+output_features:
+  - name: anomaly
+    type: anomaly
+    loss:
+      type: deep_sad
+      # eta: weight for the labeled anomaly repulsion term.
+      # Labeled anomalies (target=1 in the dataset) are pushed away from the hypersphere
+      # centre while normal/unlabeled samples are pulled toward it.
+      eta: 1.0
+
+trainer:
+  epochs: 20
+  learning_rate: 0.001
+
+combiner:
+  type: concat
+  fc_layers:
+    - output_size: 64
+    - output_size: 32
+
+# Deep SAD is semi-supervised: include a small fraction of labeled anomaly rows
+# in the training set with the anomaly column set to 1. All other training rows
+# should have anomaly=0 (normal) or anomaly=-1 (unlabeled/ignored).
+# labeled_anomalies_ratio: 0.1  # approximately 10% of training rows are labeled anomalies

examples/anomaly_detection/config_deep_svdd.yaml

@@ -0,0 +1,38 @@
+model_type: ecd
+
+input_features:
+  - name: sensor_a
+    type: number
+    preprocessing:
+      normalization: zscore
+  - name: sensor_b
+    type: number
+    preprocessing:
+      normalization: zscore
+  - name: sensor_c
+    type: number
+    preprocessing:
+      normalization: zscore
+  - name: timestamp_hour
+    type: number
+    preprocessing:
+      normalization: zscore
+
+output_features:
+  - name: anomaly
+    type: anomaly
+    loss:
+      type: deep_svdd
+      # nu: fraction of training samples allowed outside the hypersphere (soft-boundary).
+      # Set to 0 for hard-boundary SVDD where all points are pulled toward centre c.
+      nu: 0.1
+
+trainer:
+  epochs: 20
+  learning_rate: 0.001
+
+combiner:
+  type: concat
+  fc_layers:
+    - output_size: 64
+    - output_size: 32

examples/anomaly_detection/config_drocc.yaml

@@ -0,0 +1,41 @@
+model_type: ecd
+
+input_features:
+  - name: sensor_a
+    type: number
+    preprocessing:
+      normalization: zscore
+  - name: sensor_b
+    type: number
+    preprocessing:
+      normalization: zscore
+  - name: sensor_c
+    type: number
+    preprocessing:
+      normalization: zscore
+  - name: timestamp_hour
+    type: number
+    preprocessing:
+      normalization: zscore
+
+output_features:
+  - name: anomaly
+    type: anomaly
+    loss:
+      type: drocc
+      # perturbation_strength: magnitude of adversarial perturbations applied to prevent
+      # hypersphere collapse (all representations converging to center c).
+      # Typical range: 0.01–0.5. Increase if the model collapses early.
+      perturbation_strength: 0.1
+      # num_perturbation_steps: gradient ascent steps for generating each perturbation.
+      num_perturbation_steps: 5
+
+trainer:
+  epochs: 20
+  learning_rate: 0.001
+
+combiner:
+  type: concat
+  fc_layers:
+    - output_size: 64
+    - output_size: 32

examples/anomaly_detection/train.py

@@ -0,0 +1,190 @@
+# Colab: !pip install ludwig
+"""Anomaly detection with Ludwig using Deep SVDD, Deep SAD, and DROCC losses.
+
+Generates synthetic sensor data, trains all three model variants, evaluates on a
+held-out test set containing both normal and anomalous samples, and prints an AUC-ROC
+comparison table.
+
+Usage:
+    python train.py
+"""
+
+import numpy as np
+import pandas as pd
+from sklearn.metrics import roc_auc_score
+
+from ludwig.api import LudwigModel
+
+# ---------------------------------------------------------------------------
+# 1. Generate synthetic sensor data
+# ---------------------------------------------------------------------------
+
+RNG = np.random.default_rng(42)
+
+N_NORMAL = 800
+N_ANOMALY = 200
+
+# Normal samples: Gaussian near origin
+normal = pd.DataFrame(
+    {
+        "sensor_a": RNG.normal(0.0, 1.0, N_NORMAL),
+        "sensor_b": RNG.normal(0.0, 1.0, N_NORMAL),
+        "sensor_c": RNG.normal(0.0, 1.0, N_NORMAL),
+        "timestamp_hour": RNG.integers(0, 24, N_NORMAL).astype(float),
+        "anomaly": 0.0,
+    }
+)
+
+# Anomalous samples: large offset from origin
+anomalous = pd.DataFrame(
+    {
+        "sensor_a": RNG.normal(6.0, 1.0, N_ANOMALY),
+        "sensor_b": RNG.normal(6.0, 1.0, N_ANOMALY),
+        "sensor_c": RNG.normal(6.0, 1.0, N_ANOMALY),
+        "timestamp_hour": RNG.integers(0, 24, N_ANOMALY).astype(float),
+        "anomaly": 1.0,
+    }
+)
+
+all_data = pd.concat([normal, anomalous], ignore_index=True)
+
+# Train split: ONLY normal samples (anomaly detection is unsupervised)
+# Val split: mix of normal and anomalous for threshold selection
+# Test split: mix for final evaluation
+
+normal_idx = all_data[all_data["anomaly"] == 0].index.tolist()
+anomaly_idx = all_data[all_data["anomaly"] == 1].index.tolist()
+
+RNG.shuffle(normal_idx)
+n_train = int(0.7 * len(normal_idx))
+n_val = int(0.15 * len(normal_idx))
+
+train_idx = normal_idx[:n_train]
+val_normal_idx = normal_idx[n_train : n_train + n_val]
+test_normal_idx = normal_idx[n_train + n_val :]
+
+RNG.shuffle(anomaly_idx)
+n_val_anom = len(anomaly_idx) // 2
+val_anom_idx = anomaly_idx[:n_val_anom]
+test_anom_idx = anomaly_idx[n_val_anom:]
+
+# Assign split column: 0=train, 1=val, 2=test
+all_data["split"] = -1
+all_data.loc[train_idx, "split"] = 0
+all_data.loc[val_normal_idx, "split"] = 1
+all_data.loc[val_anom_idx, "split"] = 1
+all_data.loc[test_normal_idx, "split"] = 2
+all_data.loc[test_anom_idx, "split"] = 2
+
+train_df = all_data[all_data["split"] == 0].copy()
+test_df = all_data[all_data["split"].isin([1, 2])].copy()
+
+train_df.to_csv("/tmp/sensors_train.csv", index=False)
+test_df.to_csv("/tmp/sensors_test.csv", index=False)
+
+print(f"Train samples: {len(train_df)}  (all normal)")
+print(f"Test  samples: {len(test_df)}  ({(test_df['anomaly'] == 1).sum()} anomalous)")
+
+# ---------------------------------------------------------------------------
+# 2. Helper: build Ludwig config dict
+# ---------------------------------------------------------------------------
+
+INPUT_FEATURES = [
+    {"name": "sensor_a", "type": "number", "preprocessing": {"normalization": "zscore"}},
+    {"name": "sensor_b", "type": "number", "preprocessing": {"normalization": "zscore"}},
+    {"name": "sensor_c", "type": "number", "preprocessing": {"normalization": "zscore"}},
+    {"name": "timestamp_hour", "type": "number", "preprocessing": {"normalization": "zscore"}},
+]
+
+COMBINER = {
+    "type": "concat",
+    "fc_layers": [{"output_size": 64}, {"output_size": 32}],
+}
+
+TRAINER = {"epochs": 20, "learning_rate": 0.001}
+
+
+def make_config(loss: dict) -> dict:
+    return {
+        "model_type": "ecd",
+        "input_features": INPUT_FEATURES,
+        "output_features": [
+            {
+                "name": "anomaly",
+                "type": "anomaly",
+                "loss": loss,
+            }
+        ],
+        "combiner": COMBINER,
+        "trainer": TRAINER,
+    }
+
+
+CONFIGS = {
+    "Deep SVDD": make_config({"type": "deep_svdd", "nu": 0.1}),
+    "Deep SAD": make_config({"type": "deep_sad", "eta": 1.0}),
+    "DROCC": make_config({"type": "drocc", "perturbation_strength": 0.1, "num_perturbation_steps": 5}),
+}
+
+# Deep SAD: inject ~10% labeled anomalies into the training set
+N_LABELED = max(1, int(0.1 * len(train_df)))
+labeled_anom = anomalous.sample(n=N_LABELED, random_state=0).copy()
+labeled_anom["split"] = 0
+sad_train_df = pd.concat([train_df, labeled_anom], ignore_index=True)
+
+# ---------------------------------------------------------------------------
+# 3. Train and evaluate each variant
+# ---------------------------------------------------------------------------
+
+results_table = []
+
+for method_name, config in CONFIGS.items():
+    print(f"\n{'=' * 60}")
+    print(f"Training: {method_name}")
+    print("=" * 60)
+
+    train_data = sad_train_df if method_name == "Deep SAD" else train_df
+
+    model = LudwigModel(config, logging_level=30)  # WARNING level
+    train_stats, _, _ = model.train(dataset=train_data)
+
+    predictions, _ = model.predict(dataset=test_df)
+
+    score_col = "anomaly_anomaly_score_predictions"
+    scores = predictions[score_col].values
+    true_labels = test_df["anomaly"].values
+
+    auc = roc_auc_score(true_labels, scores)
+
+    # Separation ratio: mean anomaly score / mean normal score
+    normal_scores = scores[true_labels == 0]
+    anom_scores = scores[true_labels == 1]
+    sep_ratio = anom_scores.mean() / (normal_scores.mean() + 1e-9)
+
+    results_table.append(
+        {
+            "Method": method_name,
+            "AUC-ROC": round(auc, 4),
+            "Mean normal score": round(float(normal_scores.mean()), 4),
+            "Mean anomaly score": round(float(anom_scores.mean()), 4),
+            "Separation ratio": round(float(sep_ratio), 2),
+        }
+    )
+    print(f"  AUC-ROC:          {auc:.4f}")
+    print(f"  Mean normal score: {normal_scores.mean():.4f}")
+    print(f"  Mean anomaly score:{anom_scores.mean():.4f}")
+    print(f"  Separation ratio:  {sep_ratio:.2f}x")
+
+# ---------------------------------------------------------------------------
+# 4. Print summary table
+# ---------------------------------------------------------------------------
+
+results_df = pd.DataFrame(results_table)
+
+print("\n")
+print("=" * 70)
+print("ANOMALY DETECTION — SUMMARY")
+print("=" * 70)
+print(results_df.to_string(index=False))
+print("=" * 70)
+print("\nHigher AUC-ROC and separation ratio indicate better discrimination between normal and anomalous samples.")

examples/calibration/README.md

@@ -0,0 +1,22 @@
+# Calibration Examples
+
+Drawing on the methods in
+On Calibration of Modern Neural Networks (Chuan Guo, Geoff Pleiss, Yu Sun, Kilian Q. Weinberger), Ludwig supports
+temperature scaling for binary and category output features. Temperature scaling brings a model’s output probabilities
+closer to the true likelihood while preserving the same accuracy and top k predictions.
+
+To enable calibration, add calibration: True to any binary or category feature:
+
+```
+output_features:
+ - name: Cover_Type
+   type: category
+   calibration: True
+```
+
+With calibration enabled, Ludwig will attempt to find a scale factor (temperature) which will bring the probabilities
+closer to the true likelihoods using the validation set. This calibration phase is run after training is complete. If
+no validation set is provided, the training set is used for calibration.
+
+To visualize the effects of calibration in Ludwig, you can use Calibration Plots, which bin the data based on model
+probability and plot the model probability (X) versus the observed rate (Y) for each bin.

examples/calibration/train_forest_cover_calibrated.py

@@ -0,0 +1,98 @@
+#!/usr/bin/env python
+
+import copy
+import logging
+import shutil
+
+import numpy as np
+import yaml
+
+import ludwig.visualize
+from ludwig.api import LudwigModel
+from ludwig.datasets import forest_cover
+
+# clean out prior results
+shutil.rmtree("./results_forest_cover", ignore_errors=True)
+shutil.rmtree("./visualizations_forest_cover", ignore_errors=True)
+
+# Download and prepare the dataset
+dataset = forest_cover.load()
+
+config_yaml = """
+input_features:
+  - name: Elevation
+    type: number
+  - name: Aspect
+    type: number
+  - name: Slope
+    type: number
+  - name: Horizontal_Distance_To_Hydrology
+    type: number
+  - name: Vertical_Distance_To_Hydrology
+    type: number
+  - name: Horizontal_Distance_To_Roadways
+    type: number
+  - name: Hillshade_9am
+    type: number
+  - name: Hillshade_Noon
+    type: number
+  - name: Hillshade_3pm
+    type: number
+  - name: Horizontal_Distance_To_Fire_Points
+    type: number
+  - name: Wilderness_Area
+    type: category
+  - name: Soil_Type
+    type: category
+output_features:
+  - name: Cover_Type
+    type: category
+combiner:
+  type: transformer
+trainer:
+  batch_size: 256
+  learning_rate: .001
+  epochs: 1
+"""
+
+uncalibrated_config = yaml.safe_load(config_yaml)
+
+scaled_config = copy.deepcopy(uncalibrated_config)
+scaled_config["output_features"][0]["calibration"] = True
+
+uncalibrated_model = LudwigModel(config=uncalibrated_config, logging_level=logging.INFO)
+uncalibrated_model.train(
+    dataset,
+    model_name="uncalibrated",
+    experiment_name="forest_cover_calibration",
+    output_directory="results_forest_cover",
+)
+
+scaled_model = LudwigModel(config=scaled_config, logging_level=logging.INFO)
+scaled_model.train(
+    dataset, model_name="scaled", experiment_name="forest_cover_calibration", output_directory="results_forest_cover"
+)
+
+# Generates predictions and performance statistics for the test set.
+uncalibrated_test_stats, uncalibrated_test_predictions, _ = uncalibrated_model.evaluate(
+    dataset, collect_predictions=True, collect_overall_stats=True
+)
+
+scaled_test_stats, scaled_test_predictions, _ = scaled_model.evaluate(
+    dataset, collect_predictions=True, collect_overall_stats=True
+)
+
+uncalibrated_probs = np.stack(uncalibrated_test_predictions["Cover_Type_probabilities"], axis=0)
+scaled_probs = np.stack(scaled_test_predictions["Cover_Type_probabilities"], axis=0)
+
+ludwig.visualize.calibration_1_vs_all(
+    probabilities_per_model=[uncalibrated_probs, scaled_probs],
+    model_names=["Uncalibrated", "Calibrated"],
+    ground_truth=dataset["Cover_Type"],
+    metadata=uncalibrated_model.training_set_metadata,
+    output_feature_name="Cover_Type",
+    top_n_classes=[7, 7],
+    labels_limit=7,
+    output_directory="visualizations_forest_cover",
+    file_format="png",
+)

examples/calibration/train_mushroom_edibility_calibrated.py

@@ -0,0 +1,124 @@
+#!/usr/bin/env python
+
+import copy
+import logging
+import shutil
+
+import numpy as np
+import yaml
+
+import ludwig.visualize
+from ludwig.api import LudwigModel
+from ludwig.datasets import mushroom_edibility
+
+# clean out prior results
+shutil.rmtree("./results_mushroom_edibility", ignore_errors=True)
+shutil.rmtree("./visualizations_mushroom_edibility", ignore_errors=True)
+
+# Download and prepare the dataset
+dataset = mushroom_edibility.load()
+
+# This dataset has no split, so add a split column
+dataset.split = np.random.choice(3, len(dataset), p=(0.7, 0.1, 0.2))
+
+config_yaml = """
+input_features:
+  - name: cap-shape
+    type: category
+  - name: cap-surface
+    type: category
+  - name: cap-color
+    type: category
+  - name: bruises?
+    type: category
+  - name: odor
+    type: category
+  - name: gill-attachment
+    type: category
+  - name: gill-spacing
+    type: category
+  - name: gill-size
+    type: category
+  - name: gill-color
+    type: category
+  - name: stalk-shape
+    type: category
+  - name: stalk-root
+    type: category
+  - name: stalk-surface-above-ring
+    type: category
+  - name: stalk-surface-below-ring
+    type: category
+  - name: stalk-color-above-ring
+    type: category
+  - name: stalk-color-below-ring
+    type: category
+  - name: veil-type
+    type: category
+  - name: veil-color
+    type: category
+  - name: ring-number
+    type: category
+  - name: ring-type
+    type: category
+  - name: spore-print-color
+    type: category
+  - name: population
+    type: category
+  - name: habitat
+    type: category
+output_features:
+  - name: class
+    type: category
+combiner:
+  type: concat
+trainer:
+  batch_size: 256
+  learning_rate: .0001
+  epochs: 10
+"""
+
+uncalibrated_config = yaml.safe_load(config_yaml)
+
+scaled_config = copy.deepcopy(uncalibrated_config)
+scaled_config["output_features"][0]["calibration"] = True
+
+uncalibrated_model = LudwigModel(config=uncalibrated_config, logging_level=logging.INFO)
+uncalibrated_model.train(
+    dataset,
+    model_name="uncalibrated",
+    experiment_name="mushroom_edibility_calibration",
+    output_directory="results_mushroom_edibility",
+)
+
+scaled_model = LudwigModel(config=scaled_config, logging_level=logging.INFO)
+scaled_model.train(
+    dataset,
+    model_name="scaled",
+    experiment_name="mushroom_edibility_calibration",
+    output_directory="results_mushroom_edibility",
+)
+
+# Generates predictions and performance statistics for the test set.
+uncalibrated_test_stats, uncalibrated_test_predictions, _ = uncalibrated_model.evaluate(
+    dataset, collect_predictions=True, collect_overall_stats=True
+)
+
+scaled_test_stats, scaled_test_predictions, _ = scaled_model.evaluate(
+    dataset, collect_predictions=True, collect_overall_stats=True
+)
+
+uncalibrated_probs = np.stack(uncalibrated_test_predictions["class_probabilities"], axis=0)
+scaled_probs = np.stack(scaled_test_predictions["class_probabilities"], axis=0)
+
+ludwig.visualize.calibration_1_vs_all(
+    probabilities_per_model=[uncalibrated_probs, scaled_probs],
+    model_names=["Uncalibrated", "Calibrated"],
+    ground_truth=dataset["class"],
+    metadata=uncalibrated_model.training_set_metadata,
+    output_feature_name="class",
+    top_n_classes=[3, 3],
+    labels_limit=3,
+    output_directory="visualizations_mushroom_edibility",
+    file_format="png",
+)

examples/class_imbalance/README.md

@@ -0,0 +1,50 @@
+# Credit Card Fraud Detection Example
+
+This API example is based on Kaggle's [Imbalanced Insurance](https://www.kaggle.com/arashnic/imbalanced-data-practice) dataset for detecting whether customers will buy vehicle insurance.
+
+### Preparatory Steps
+
+Create and download your [Kaggle API Credentials](https://github.com/Kaggle/kaggle-api#api-credentials).
+
+The Imbalanced Insurance dataset is hosted by Kaggle, and as such Ludwig will need to authenticate you through the Kaggle API to download the dataset.
+
+### Examples
+
+| File                         | Description                                                                                                    |
+| ---------------------------- | -------------------------------------------------------------------------------------------------------------- |
+| model_training.py            | Demonstrates the use of oversampling by training two different models: one with no oversampling, and one with. |
+| model_training_results.ipynb | Example for extracting training statistics and generating visualizations.                                      |
+
+Enter `python model_training.py` will train a standard model with no class balancing and a balanced model with class balancing applied. Results of model training will be stored in this location.
+
+```
+./results/
+    balance_example_standard_model/
+    balance_example_balanced_model/
+```
+
+The only difference between these two models is that the balanced model uses a small amount of oversampling in addition to the other configuration parameters.
+The way this is done is by specifying the ratio that you want the minority class to have in relation to the majority class.
+For instance, if you specify 0.5 for the `oversample_minority` preprocessing parameter, the minority class will be oversampled until it makes up 50% of the majority class.
+In this example, we had an imbalance where the minority class was 19% of the majority class in size. We decided that we wanted to increase that to 26%.
+Though this doesn't seem like much, it is enough to get some small performance improvements without experiencing performance degradation due to over-fitting.
+
+Here are the performance differences in the two models followed by some plots showing different metrics over training:
+
+|  Metric  | Standard Model | Balanced Model |
+| :------: | :------------: | :------------: |
+|   Loss   |     0.3649     |     0.2758     |
+| Accuracy |     0.7732     |     0.8237     |
+| ROC AUC  |     0.8533     |     0.8660     |
+
+Here are the learning curve plots from both models:
+
+![](../images/balance_example_accuracy_curves.png)
+
+![](../images/balance_example_roc_auc_curves.png)
+
+Here is the comparison of model performances on ROC_AUC and Accuracy:
+
+![](../images/compare_performance_Response.png)
+
+The creation of the learning curves is demonstrated in the Jupyter notebook `model_training_results.ipynb`. The comparison plot was generated using the ludwig visualize [compare performance](https://ludwig-ai.github.io/ludwig-docs/0.4/user_guide/visualizations/#compare-performance) command.

examples/class_imbalance/balanced_model_config.yaml

@@ -0,0 +1,34 @@
+input_features:
+  - name: Gender
+    type: category
+  - name: Age
+    type: number
+  - name: Driving_License
+    type: binary
+  - name: Region_Code
+    type: number
+  - name: Previously_Insured
+    type: binary
+  - name: Vehicle_Age
+    type: category
+  - name: Vehicle_Damage
+    type: category
+  - name: Annual_Premium
+    type: number
+  - name: Policy_Sales_Channel
+    type: number
+  - name: Vintage
+    type: number
+output_features:
+  - name: Response
+    type: binary
+preprocessing:
+  oversample_minority: 0.26
+trainer:
+  learning_rate: 0.0001
+  learning_rate_scheduler:
+    decay: exponential
+    decay_rate: 0.9
+    decay_steps: 30000
+    staircase: True
+  epochs: 50

examples/class_imbalance/model_training.py

@@ -0,0 +1,57 @@
+#!/usr/bin/env python
+
+# # Class Imbalance Model Training Example
+#
+# This example trains a model utilizing a standard config, and then a config using oversampling
+
+import logging
+import shutil
+
+# Import required libraries
+from ludwig.api import LudwigModel
+from ludwig.datasets import imbalanced_insurance
+from ludwig.visualize import compare_performance
+
+# clean out old results
+shutil.rmtree("./results", ignore_errors=True)
+shutil.rmtree("./visualizations", ignore_errors=True)
+
+# list models to train
+list_of_model_ids = ["standard_model", "balanced_model"]
+list_of_eval_stats = []
+
+training_set, val_set, test_set = imbalanced_insurance.load()
+
+# Train models
+for model_id in list_of_model_ids:
+    print(">>>> training: ", model_id)
+
+    # Define Ludwig model object that drive model training
+    model = LudwigModel(config=model_id + "_config.yaml", logging_level=logging.WARN)
+
+    # initiate model training
+    train_stats, _, _ = model.train(
+        training_set=training_set,
+        validation_set=val_set,
+        test_set=test_set,
+        experiment_name="balance_example",
+        model_name=model_id,
+        skip_save_model=True,
+    )
+
+    # evaluate model on test_set
+    eval_stats, _, _ = model.evaluate(test_set)
+
+    # save eval stats for later use
+    list_of_eval_stats.append(eval_stats)
+
+    print(">>>>>>> completed: ", model_id, "\n")
+
+
+compare_performance(
+    list_of_eval_stats,
+    "Response",
+    model_names=list_of_model_ids,
+    output_directory="./visualizations",
+    file_format="png",
+)

examples/class_imbalance/standard_model_config.yaml

@@ -0,0 +1,32 @@
+input_features:
+  - name: Gender
+    type: category
+  - name: Age
+    type: number
+  - name: Driving_License
+    type: binary
+  - name: Region_Code
+    type: number
+  - name: Previously_Insured
+    type: binary
+  - name: Vehicle_Age
+    type: category
+  - name: Vehicle_Damage
+    type: category
+  - name: Annual_Premium
+    type: number
+  - name: Policy_Sales_Channel
+    type: number
+  - name: Vintage
+    type: number
+output_features:
+  - name: Response
+    type: binary
+trainer:
+  learning_rate: 0.0001
+  learning_rate_scheduler:
+    decay: exponential
+    decay_rate: 0.9
+    decay_steps: 30000
+    staircase: True
+  epochs: 50

examples/forecasting/README.md

@@ -0,0 +1,3 @@
+- Download and unpack hourly weather data from https://www.kaggle.com/selfishgene/historical-hourly-weather-data
+- `ludwig train --config config.yaml --dataset temperature.csv`
+- `ludwig forecast -n 10 --model_path results/experiment_run/model --dataset temperature.csv`

examples/forecasting/config.yaml

@@ -0,0 +1,20 @@
+input_features:
+  - name: Seattle
+    type: timeseries
+    preprocessing:
+      window_size: 10
+    encoder:
+      type: passthrough
+output_features:
+  - name: Seattle_next
+    type: timeseries
+    column: Seattle
+    preprocessing:
+      horizon: 2
+combiner:
+  type: concat
+  flatten_inputs: true
+preprocessing:
+  split:
+    type: datetime
+    column: datetime

examples/getting_started/rotten_tomatoes.yaml

@@ -0,0 +1,20 @@
+input_features:
+  - name: genres
+    type: set
+    preprocessing:
+      tokenizer: comma
+  - name: content_rating
+    type: category
+  - name: top_critic
+    type: binary
+  - name: runtime
+    type: number
+  - name: review_content
+    type: text
+    encoder:
+      type: embed
+output_features:
+  - name: recommended
+    type: binary
+trainer:
+  epochs: 3

examples/getting_started/run.sh

@@ -0,0 +1,20 @@
+#!/usr/bin/env bash
+
+# Fail fast if an error occurs
+set -e
+
+# Get the directory of this script, which contains the config file
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
+
+# Download the data
+wget https://ludwig.ai/latest/data/rotten_tomatoes.csv
+wget https://ludwig.ai/latest/data/rotten_tomatoes_test.csv
+
+# Check the first 5 rows
+head -n 5 rotten_tomatoes.csv
+
+# Train
+ludwig train --config ${SCRIPT_DIR}/rotten_tomatoes.yaml --dataset rotten_tomatoes.csv
+
+# Predict and Evaluate
+ludwig predict --model_path results/experiment_run/model --dataset rotten_tomatoes_test.csv

examples/hypernetwork/README.md

@@ -0,0 +1,73 @@
+# HyperNetworkCombiner: Conditional Feature Processing
+
+> **Note:** This example requires PR #4092 to be merged into Ludwig, or `pip install ludwig` >= 0.14.
+
+[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ludwig-ai/ludwig/blob/main/examples/hypernetwork/hypernetwork.ipynb)
+
+## What the HyperNetworkCombiner does differently
+
+Most combiners — including the default `concat` combiner — treat all input features
+symmetrically: they encode each feature independently and then merge the resulting
+vectors (by concatenation, attention, or summation). The merged representation is
+the same *kind* of computation regardless of what any individual feature says.
+
+The `hypernetwork` combiner breaks this symmetry. One feature, called the
+**conditioning feature**, is fed through a small *hyper-network* that generates the
+weight matrices and biases of the fully-connected layers that process all other features.
+In other words, the conditioning feature does not just *contribute* to the prediction —
+it *rewrites the transformation* applied to every other feature before prediction happens.
+
+This is based on **HyperFusion** (arXiv 2403.13319, 2024).
+
+```
+sensor_type  ──► HyperNetwork ──► generates weights W, b
+                                        │
+sensor_a ─────────────────────► FC(W, b) ──► combined
+sensor_b ─────────────────────► FC(W, b) ──►  repr.
+sensor_c ─────────────────────► FC(W, b) ──►
+```
+
+Contrast with concat:
+
+```
+sensor_type ──►  encoder  ──┐
+sensor_a    ──►  encoder  ──┤
+sensor_b    ──►  encoder  ──┼──► concat ──► FC ──► output
+sensor_c    ──►  encoder  ──┘
+```
+
+With `concat`, the network learns *after* combining to react to different sensor types.
+With `hypernetwork`, the combination itself is conditioned on sensor type.
+
+## When to use it
+
+Use the `hypernetwork` combiner when:
+
+- One feature is a **context** or **mode** that fundamentally changes how other
+  features should be interpreted (sensor type, device class, environment, language).
+- The relationship between inputs and the target changes qualitatively across groups,
+  not just quantitatively.
+- You have enough training data to learn the per-context transformations (at minimum a
+  few hundred samples per conditioning category).
+
+Stick with `concat` when:
+
+- All input features contribute on equal footing.
+- The dataset is small (the hyper-network adds parameters).
+- Interpretability of the encoding step is important and you want a fixed transformation.
+
+## Files
+
+| File                       | Description                                                               |
+| -------------------------- | ------------------------------------------------------------------------- |
+| `hypernetwork.ipynb`       | End-to-end walkthrough with synthetic sensor data                         |
+| `config_concat.yaml`       | Baseline concat config                                                    |
+| `config_hypernetwork.yaml` | HyperNetworkCombiner config                                               |
+| `train_hypernetwork.py`    | Standalone script — generates data, trains both models, prints comparison |
+
+## Quick start
+
+```bash
+pip install "ludwig>=0.14"
+python train_hypernetwork.py
+```

examples/hypernetwork/config_concat.yaml

@@ -0,0 +1,31 @@
+model_type: ecd
+
+input_features:
+  - name: sensor_a
+    type: number
+    preprocessing:
+      normalization: zscore
+  - name: sensor_b
+    type: number
+    preprocessing:
+      normalization: zscore
+  - name: sensor_c
+    type: number
+    preprocessing:
+      normalization: zscore
+  - name: sensor_type
+    type: category
+
+output_features:
+  - name: anomaly
+    type: binary
+
+combiner:
+  type: concat
+  fc_layers:
+    - output_size: 128
+    - output_size: 64
+
+trainer:
+  epochs: 30
+  learning_rate: 0.001

examples/hypernetwork/config_hypernetwork.yaml

@@ -0,0 +1,31 @@
+model_type: ecd
+
+input_features:
+  - name: sensor_a
+    type: number
+    preprocessing:
+      normalization: zscore
+  - name: sensor_b
+    type: number
+    preprocessing:
+      normalization: zscore
+  - name: sensor_c
+    type: number
+    preprocessing:
+      normalization: zscore
+  - name: sensor_type
+    type: category
+
+output_features:
+  - name: anomaly
+    type: binary
+
+combiner:
+  type: hypernetwork
+  hidden_size: 128
+  hyper_hidden_size: 64
+  output_size: 128
+
+trainer:
+  epochs: 30
+  learning_rate: 0.001

examples/hypernetwork/hypernetwork.ipynb

@@ -0,0 +1,427 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "a1b2c3d4-0001-0000-0000-000000000001",
+   "metadata": {},
+   "source": [
+    "# HyperNetworkCombiner: Conditional Feature Processing\n",
+    "\n",
+    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ludwig-ai/ludwig/blob/main/examples/hypernetwork/hypernetwork.ipynb)\n",
+    "\n",
+    "> **Note:** This notebook requires **Ludwig >= 0.14** (PR #4092). The `hypernetwork`\n",
+    "> combiner type is not available in earlier releases. Install with:\n",
+    "> `pip install \"ludwig>=0.14\"`\n",
+    "\n",
+    "This notebook demonstrates the `HyperNetworkCombiner`, which lets one feature\n",
+    "(**the conditioning feature**) generate the weights of the layers that process all\n",
+    "other features — rather than simply concatenating everyone together.\n",
+    "\n",
+    "Based on **HyperFusion** ([arXiv 2403.13319](https://arxiv.org/abs/2403.13319), 2024).\n",
+    "\n",
+    "**What we cover:**\n",
+    "\n",
+    "1. Why concatenation is not always enough\n",
+    "2. Generating a synthetic multi-modal sensor dataset\n",
+    "3. Baseline: concat combiner\n",
+    "4. HyperNetworkCombiner\n",
+    "5. Comparing results and understanding why hypernetwork wins"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a1b2c3d4-0002-0000-0000-000000000002",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!pip install \"ludwig>=0.14\" --quiet"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a1b2c3d4-0003-0000-0000-000000000003",
+   "metadata": {},
+   "source": [
+    "## The problem: context-dependent features\n",
+    "\n",
+    "Imagine a network of industrial sensors. Each sensor reports three readings\n",
+    "(`sensor_a`, `sensor_b`, `sensor_c`), but every sensor belongs to one of three\n",
+    "measurement types: **temperature**, **pressure**, or **humidity**.\n",
+    "\n",
+    "The catch: **the same numerical reading means something completely different**\n",
+    "depending on the sensor type.\n",
+    "\n",
+    "- For a **temperature** sensor, `sensor_a = 3.0` is an anomaly (overheating).\n",
+    "- For a **pressure** sensor, `sensor_a = 3.0` is perfectly normal.\n",
+    "- For a **humidity** sensor, the anomaly rule involves the *sum* of all three readings.\n",
+    "\n",
+    "A concat combiner encodes all four features independently and then stitches them\n",
+    "together. The network has to learn — **after** the concatenation — to undo the mixing\n",
+    "and apply type-specific logic. This is hard because the critical signal (`sensor_type`)\n",
+    "is buried in a shared representation alongside the numerical readings.\n",
+    "\n",
+    "The `hypernetwork` combiner solves this directly:\n",
+    "\n",
+    "```\n",
+    "sensor_type  ──► HyperNetwork ──► generates weights W, b\n",
+    "                                       │\n",
+    "sensor_a ────────────────────► FC(W, b) ──►\n",
+    "sensor_b ────────────────────► FC(W, b) ──►  combined repr.\n",
+    "sensor_c ────────────────────► FC(W, b) ──►\n",
+    "```\n",
+    "\n",
+    "`sensor_type` does not contribute a feature vector — it **rewrites the entire\n",
+    "transformation** applied to the numerical sensors."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a1b2c3d4-0004-0000-0000-000000000004",
+   "metadata": {},
+   "source": [
+    "## Dataset\n",
+    "\n",
+    "We generate a synthetic dataset with three sensor types. Each type has its own\n",
+    "normal operating range and its own anomaly rule, making the sensor type an\n",
+    "essential piece of context for correct classification."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a1b2c3d4-0005-0000-0000-000000000005",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "import pandas as pd\n",
+    "\n",
+    "RNG = np.random.default_rng(42)\n",
+    "\n",
+    "N_PER_TYPE = 600\n",
+    "SENSOR_TYPES = [\"temperature\", \"pressure\", \"humidity\"]\n",
+    "\n",
+    "\n",
+    "def make_samples(sensor_type: str, n: int, rng: np.random.Generator) -> pd.DataFrame:\n",
+    "    \"\"\"Generate n samples for a single sensor type with type-specific anomaly rules.\"\"\"\n",
+    "    if sensor_type == \"temperature\":\n",
+    "        # Normal: readings near 0; anomaly: sensor_a > 2.5 (overheating)\n",
+    "        sensor_a = rng.normal(0.0, 1.0, n)\n",
+    "        sensor_b = rng.normal(0.0, 1.0, n)\n",
+    "        sensor_c = rng.normal(0.0, 1.0, n)\n",
+    "        anomaly = (sensor_a > 2.5).astype(int)\n",
+    "    elif sensor_type == \"pressure\":\n",
+    "        # Normal: readings near 1; anomaly: sensor_b drops below -0.5 (leak)\n",
+    "        sensor_a = rng.normal(1.0, 0.8, n)\n",
+    "        sensor_b = rng.normal(1.0, 0.8, n)\n",
+    "        sensor_c = rng.normal(1.0, 0.8, n)\n",
+    "        anomaly = (sensor_b < -0.5).astype(int)\n",
+    "    else:  # humidity\n",
+    "        # Normal: readings near -1; anomaly: combined level exceeds threshold\n",
+    "        sensor_a = rng.normal(-1.0, 0.9, n)\n",
+    "        sensor_b = rng.normal(-1.0, 0.9, n)\n",
+    "        sensor_c = rng.normal(-1.0, 0.9, n)\n",
+    "        anomaly = ((sensor_a + sensor_b + sensor_c) > 0).astype(int)\n",
+    "\n",
+    "    return pd.DataFrame(\n",
+    "        {\n",
+    "            \"sensor_a\": sensor_a,\n",
+    "            \"sensor_b\": sensor_b,\n",
+    "            \"sensor_c\": sensor_c,\n",
+    "            \"sensor_type\": sensor_type,\n",
+    "            \"anomaly\": anomaly,\n",
+    "        }\n",
+    "    )\n",
+    "\n",
+    "\n",
+    "frames = [make_samples(t, N_PER_TYPE, RNG) for t in SENSOR_TYPES]\n",
+    "df = pd.concat(frames, ignore_index=True).sample(frac=1, random_state=42).reset_index(drop=True)\n",
+    "\n",
+    "# Train / validation / test split  (70 / 15 / 15)\n",
+    "n = len(df)\n",
+    "split = np.full(n, 2, dtype=int)  # default: test\n",
+    "idx = np.arange(n)\n",
+    "RNG.shuffle(idx)\n",
+    "split[idx[: int(0.70 * n)]] = 0\n",
+    "split[idx[int(0.70 * n) : int(0.85 * n)]] = 1\n",
+    "df[\"split\"] = split\n",
+    "\n",
+    "print(f\"Total rows: {n}\")\n",
+    "print(f\"Overall anomaly rate: {df['anomaly'].mean():.1%}\")\n",
+    "print()\n",
+    "print(\"Anomaly rate by sensor type:\")\n",
+    "print(df.groupby(\"sensor_type\")[\"anomaly\"].mean().rename(\"anomaly_rate\"))\n",
+    "print()\n",
+    "df.head(10)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a1b2c3d4-0006-0000-0000-000000000006",
+   "metadata": {},
+   "source": [
+    "Notice that the three sensor types have **different anomaly rates** and, more\n",
+    "importantly, the anomaly rules are structurally different. The same value of\n",
+    "`sensor_a = 3.0` triggers an anomaly for `temperature` but not for `pressure`.\n",
+    "A model that treats all features symmetrically will struggle with this."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a1b2c3d4-0007-0000-0000-000000000007",
+   "metadata": {},
+   "source": [
+    "## Baseline: concat combiner\n",
+    "\n",
+    "The default Ludwig combiner concatenates all encoder outputs and passes them\n",
+    "through fully-connected layers. `sensor_type` is just another input — its\n",
+    "embedding is concatenated alongside the numerical sensor values."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a1b2c3d4-0008-0000-0000-000000000008",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import yaml\n",
+    "\n",
+    "from ludwig.api import LudwigModel\n",
+    "\n",
+    "config_concat_str = \"\"\"\n",
+    "model_type: ecd\n",
+    "\n",
+    "input_features:\n",
+    "  - name: sensor_a\n",
+    "    type: number\n",
+    "    preprocessing:\n",
+    "      normalization: zscore\n",
+    "  - name: sensor_b\n",
+    "    type: number\n",
+    "    preprocessing:\n",
+    "      normalization: zscore\n",
+    "  - name: sensor_c\n",
+    "    type: number\n",
+    "    preprocessing:\n",
+    "      normalization: zscore\n",
+    "  - name: sensor_type\n",
+    "    type: category\n",
+    "\n",
+    "output_features:\n",
+    "  - name: anomaly\n",
+    "    type: binary\n",
+    "\n",
+    "combiner:\n",
+    "  type: concat\n",
+    "  fc_layers:\n",
+    "    - output_size: 128\n",
+    "    - output_size: 64\n",
+    "\n",
+    "trainer:\n",
+    "  epochs: 30\n",
+    "  learning_rate: 0.001\n",
+    "\"\"\"\n",
+    "\n",
+    "config_concat = yaml.safe_load(config_concat_str)\n",
+    "\n",
+    "model_concat = LudwigModel(config_concat, logging_level=30)\n",
+    "model_concat.train(dataset=df)\n",
+    "\n",
+    "test_df = df[df[\"split\"] == 2].copy()\n",
+    "preds_concat, _ = model_concat.predict(dataset=test_df)\n",
+    "\n",
+    "acc_concat = (preds_concat[\"anomaly_predictions\"].values == test_df[\"anomaly\"].values).mean()\n",
+    "print(f\"Concat combiner — test accuracy: {acc_concat:.4f}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a1b2c3d4-0009-0000-0000-000000000009",
+   "metadata": {},
+   "source": [
+    "## HyperNetworkCombiner\n",
+    "\n",
+    "Now we switch to `type: hypernetwork`. The combiner reads `sensor_type` (the last\n",
+    "feature listed in `input_features`) through a hyper-network and uses the output to\n",
+    "generate the weight matrix and bias of the layer that processes `sensor_a`,\n",
+    "`sensor_b`, and `sensor_c`.\n",
+    "\n",
+    "Key parameters:\n",
+    "\n",
+    "| Parameter | Role |\n",
+    "|---|---|\n",
+    "| `hidden_size` | Size of the main processing layer |\n",
+    "| `hyper_hidden_size` | Hidden size of the hyper-network itself |\n",
+    "| `output_size` | Dimension of the combined representation passed to decoders |"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a1b2c3d4-0010-0000-0000-000000000010",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "config_hypernetwork_str = \"\"\"\n",
+    "model_type: ecd\n",
+    "\n",
+    "input_features:\n",
+    "  - name: sensor_a\n",
+    "    type: number\n",
+    "    preprocessing:\n",
+    "      normalization: zscore\n",
+    "  - name: sensor_b\n",
+    "    type: number\n",
+    "    preprocessing:\n",
+    "      normalization: zscore\n",
+    "  - name: sensor_c\n",
+    "    type: number\n",
+    "    preprocessing:\n",
+    "      normalization: zscore\n",
+    "  - name: sensor_type\n",
+    "    type: category\n",
+    "\n",
+    "output_features:\n",
+    "  - name: anomaly\n",
+    "    type: binary\n",
+    "\n",
+    "combiner:\n",
+    "  type: hypernetwork\n",
+    "  hidden_size: 128\n",
+    "  hyper_hidden_size: 64\n",
+    "  output_size: 128\n",
+    "\n",
+    "trainer:\n",
+    "  epochs: 30\n",
+    "  learning_rate: 0.001\n",
+    "\"\"\"\n",
+    "\n",
+    "config_hypernetwork = yaml.safe_load(config_hypernetwork_str)\n",
+    "\n",
+    "model_hypernetwork = LudwigModel(config_hypernetwork, logging_level=30)\n",
+    "model_hypernetwork.train(dataset=df)\n",
+    "\n",
+    "preds_hyper, _ = model_hypernetwork.predict(dataset=test_df)\n",
+    "\n",
+    "acc_hyper = (preds_hyper[\"anomaly_predictions\"].values == test_df[\"anomaly\"].values).mean()\n",
+    "print(f\"HyperNetworkCombiner — test accuracy: {acc_hyper:.4f}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a1b2c3d4-0011-0000-0000-000000000011",
+   "metadata": {},
+   "source": [
+    "## Comparison"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a1b2c3d4-0012-0000-0000-000000000012",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import matplotlib.pyplot as plt\n",
+    "\n",
+    "# Per-type breakdown\n",
+    "rows = []\n",
+    "for stype in SENSOR_TYPES:\n",
+    "    mask = test_df[\"sensor_type\"] == stype\n",
+    "    true = test_df.loc[mask, \"anomaly\"].values\n",
+    "\n",
+    "    acc_c = (preds_concat.loc[mask, \"anomaly_predictions\"].values == true).mean()\n",
+    "    acc_h = (preds_hyper.loc[mask, \"anomaly_predictions\"].values == true).mean()\n",
+    "    rows.append({\"sensor_type\": stype, \"concat\": round(acc_c, 4), \"hypernetwork\": round(acc_h, 4)})\n",
+    "\n",
+    "rows.append({\"sensor_type\": \"OVERALL\", \"concat\": round(acc_concat, 4), \"hypernetwork\": round(acc_hyper, 4)})\n",
+    "\n",
+    "results_df = pd.DataFrame(rows)\n",
+    "print(results_df.to_string(index=False))\n",
+    "\n",
+    "# Bar chart\n",
+    "fig, ax = plt.subplots(figsize=(8, 4))\n",
+    "x = np.arange(len(rows))\n",
+    "width = 0.35\n",
+    "ax.bar(x - width / 2, results_df[\"concat\"], width, label=\"Concat\", color=\"steelblue\")\n",
+    "ax.bar(x + width / 2, results_df[\"hypernetwork\"], width, label=\"HyperNetwork\", color=\"darkorange\")\n",
+    "ax.set_xticks(x)\n",
+    "ax.set_xticklabels(results_df[\"sensor_type\"])\n",
+    "ax.set_ylabel(\"Test accuracy\")\n",
+    "ax.set_ylim(0.5, 1.0)\n",
+    "ax.set_title(\"Sensor anomaly detection: concat vs HyperNetworkCombiner\")\n",
+    "ax.legend()\n",
+    "plt.tight_layout()\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a1b2c3d4-0013-0000-0000-000000000013",
+   "metadata": {},
+   "source": [
+    "## Why hypernetwork wins\n",
+    "\n",
+    "### The problem with concatenation\n",
+    "\n",
+    "When we use `concat`, the model receives a vector like:\n",
+    "\n",
+    "```\n",
+    "[enc(sensor_a), enc(sensor_b), enc(sensor_c), enc(sensor_type)]\n",
+    "```\n",
+    "\n",
+    "The fully-connected layers after the concat have to learn — from scratch — that\n",
+    "`enc(sensor_type)` should *gate* the interpretation of the numerical sensors.\n",
+    "Effectively the network must implement a conditional logic as a series of\n",
+    "multiplications and additions over the entire concatenated vector. This is\n",
+    "theoretically possible but inefficient: many capacity-bearing parameters in the\n",
+    "FC layers end up implementing the routing rather than the actual anomaly detection.\n",
+    "\n",
+    "### What hypernetwork does instead\n",
+    "\n",
+    "The `HyperNetworkCombiner` separates the two roles explicitly:\n",
+    "\n",
+    "1. **Hyper-network** — a small MLP that reads `sensor_type` and emits a vector\n",
+    "   of weights `W` and biases `b`.\n",
+    "2. **Main network** — a linear layer `FC(W, b)` applied to the concatenation of\n",
+    "   `[sensor_a, sensor_b, sensor_c]` using the *dynamically generated* `W` and `b`.\n",
+    "\n",
+    "Because `W` and `b` are different for each sensor type, the transformation\n",
+    "applied to the numerical sensors is literally different per context. For\n",
+    "`temperature`, the generated `W` learns to make `sensor_a` highly predictive;\n",
+    "for `pressure`, the generated `W` shifts attention to `sensor_b`; for\n",
+    "`humidity`, it learns to combine all three.\n",
+    "\n",
+    "### When to use it\n",
+    "\n",
+    "- The conditioning feature is a **type, class, mode, or context** that changes\n",
+    "  the semantics of other features qualitatively.\n",
+    "- You have enough samples per conditioning category (roughly 200+ per class) to\n",
+    "  learn meaningful per-context transformations.\n",
+    "- The target signal requires different logic for different contexts, not just\n",
+    "  different magnitudes.\n",
+    "\n",
+    "### When to stick with concat\n",
+    "\n",
+    "- All features contribute on equal footing with no hierarchical conditioning.\n",
+    "- The dataset is very small — the hyper-network adds parameters.\n",
+    "- The extra complexity is not warranted (always start simple)."
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "name": "python",
+   "version": "3.12.0"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

examples/hypernetwork/train_hypernetwork.py

@@ -0,0 +1,160 @@
+# Colab: !pip install "ludwig>=0.14"
+"""HyperNetworkCombiner vs concat — sensor anomaly detection.
+
+Generates a synthetic multi-modal sensor dataset where the correct interpretation
+of numerical sensor readings depends entirely on the sensor type (temperature,
+pressure, or humidity). Trains a baseline concat model and a HyperNetworkCombiner
+model, then prints an accuracy comparison.
+
+NOTE: Requires ludwig >= 0.14 (PR #4092). The hypernetwork combiner is not
+available in earlier versions.
+
+Usage:
+    python train_hypernetwork.py
+"""
+
+import numpy as np
+import pandas as pd
+
+from ludwig.api import LudwigModel
+
+# ---------------------------------------------------------------------------
+# 1. Generate synthetic sensor data
+# ---------------------------------------------------------------------------
+
+RNG = np.random.default_rng(42)
+
+N_PER_TYPE = 600
+SENSOR_TYPES = ["temperature", "pressure", "humidity"]
+
+
+def make_samples(sensor_type: str, n: int, rng: np.random.Generator) -> pd.DataFrame:
+    """Generate n samples for a single sensor type.
+
+    Each type has its own 'normal' operating range and anomaly rule so that the same raw reading can mean very different
+    things depending on the type.
+    """
+    if sensor_type == "temperature":
+        # Normal: sensors cluster near (0, 0, 0); anomaly: sensor_a > 2.5
+        sensor_a = rng.normal(0.0, 1.0, n)
+        sensor_b = rng.normal(0.0, 1.0, n)
+        sensor_c = rng.normal(0.0, 1.0, n)
+        anomaly = (sensor_a > 2.5).astype(int)
+    elif sensor_type == "pressure":
+        # Normal: sensors cluster near (1, 1, 1); anomaly: sensor_b < -1.5
+        sensor_a = rng.normal(1.0, 0.8, n)
+        sensor_b = rng.normal(1.0, 0.8, n)
+        sensor_c = rng.normal(1.0, 0.8, n)
+        anomaly = (sensor_b < -0.5).astype(int)
+    else:  # humidity
+        # Normal: sensors cluster near (-1, -1, -1); anomaly: sum > 0
+        sensor_a = rng.normal(-1.0, 0.9, n)
+        sensor_b = rng.normal(-1.0, 0.9, n)
+        sensor_c = rng.normal(-1.0, 0.9, n)
+        anomaly = ((sensor_a + sensor_b + sensor_c) > 0).astype(int)
+
+    return pd.DataFrame(
+        {
+            "sensor_a": sensor_a,
+            "sensor_b": sensor_b,
+            "sensor_c": sensor_c,
+            "sensor_type": sensor_type,
+            "anomaly": anomaly,
+        }
+    )
+
+
+frames = [make_samples(t, N_PER_TYPE, RNG) for t in SENSOR_TYPES]
+df = pd.concat(frames, ignore_index=True).sample(frac=1, random_state=42).reset_index(drop=True)
+
+# Train / validation / test split  (70 / 15 / 15)
+n = len(df)
+split = np.full(n, 2, dtype=int)  # default: test
+idx = np.arange(n)
+RNG.shuffle(idx)
+split[idx[: int(0.70 * n)]] = 0
+split[idx[int(0.70 * n) : int(0.85 * n)]] = 1
+df["split"] = split
+
+print(f"Dataset: {n} rows  ({df['anomaly'].mean():.1%} anomalies)")
+print(df.groupby("sensor_type")["anomaly"].mean().rename("anomaly_rate").to_string())
+print()
+
+# ---------------------------------------------------------------------------
+# 2. Model configs
+# ---------------------------------------------------------------------------
+
+INPUT_FEATURES = [
+    {"name": "sensor_a", "type": "number", "preprocessing": {"normalization": "zscore"}},
+    {"name": "sensor_b", "type": "number", "preprocessing": {"normalization": "zscore"}},
+    {"name": "sensor_c", "type": "number", "preprocessing": {"normalization": "zscore"}},
+    {"name": "sensor_type", "type": "category"},
+]
+
+OUTPUT_FEATURES = [{"name": "anomaly", "type": "binary"}]
+
+TRAINER = {"epochs": 30, "learning_rate": 0.001}
+
+config_concat = {
+    "model_type": "ecd",
+    "input_features": INPUT_FEATURES,
+    "output_features": OUTPUT_FEATURES,
+    "combiner": {
+        "type": "concat",
+        "fc_layers": [{"output_size": 128}, {"output_size": 64}],
+    },
+    "trainer": TRAINER,
+}
+
+config_hypernetwork = {
+    "model_type": "ecd",
+    "input_features": INPUT_FEATURES,
+    "output_features": OUTPUT_FEATURES,
+    "combiner": {
+        "type": "hypernetwork",
+        "hidden_size": 128,
+        "hyper_hidden_size": 64,
+        "output_size": 128,
+    },
+    "trainer": TRAINER,
+}
+
+# ---------------------------------------------------------------------------
+# 3. Train and evaluate
+# ---------------------------------------------------------------------------
+
+results = []
+
+for label, config in [("Concat (baseline)", config_concat), ("HyperNetwork", config_hypernetwork)]:
+    print(f"{'=' * 60}")
+    print(f"Training: {label}")
+    print("=" * 60)
+
+    model = LudwigModel(config, logging_level=30)
+    results_obj = model.train(dataset=df)
+    print(f"  Saved to: {results_obj.output_directory}")
+
+    test_df = df[df["split"] == 2].copy()
+    predictions, _ = model.predict(dataset=test_df)
+
+    pred_col = "anomaly_predictions"
+    correct = (predictions[pred_col].values == test_df["anomaly"].values).mean()
+
+    results.append({"Model": label, "Test accuracy": round(float(correct), 4)})
+    print(f"  Test accuracy: {correct:.4f}\n")
+
+# ---------------------------------------------------------------------------
+# 4. Print summary
+# ---------------------------------------------------------------------------
+
+results_df = pd.DataFrame(results)
+
+print("=" * 50)
+print("SENSOR ANOMALY DETECTION — SUMMARY")
+print("=" * 50)
+print(results_df.to_string(index=False))
+print("=" * 50)
+print()
+print("The HyperNetworkCombiner lets sensor_type rewrite the")
+print("transformation applied to sensor_a/b/c rather than")
+print("just concatenating all features together.")

examples/hyperopt/README.md

@@ -0,0 +1,28 @@
+# Hyperparameter Optimization
+
+Demonstrates hyperparameter optimization using Ludwig's in-built capabilities.
+
+### Preparatory Steps
+
+- Create `data` directory
+- Download [Kaggle wine quality data set](https://www.kaggle.com/rajyellow46/wine-quality) into the `data` directory. Directory should
+  appear as follows:
+
+```
+hyperopt/
+    data/
+        winequalityN.csv
+```
+
+### Description
+
+Jupyter notebook `model_hyperopt_example.ipynb` demonstrates several hyperparameter optimization capabilities. Key features demonstrated in the notebook:
+
+- Training data is prepared for use
+- Programmatically create Ludwig config dictionary from the training data dataframe
+- Setup parameter space for hyperparameter optimization
+- Perform two hyperparameter runs
+  - Parallel workers using random search strategy
+  - Serial processing using random search strategy
+  - Parallel workers using grid search strategy (Note: takes about 35 minutes)
+- Demonstrate various Ludwig visualizations for hyperparameter optimization

examples/hyperopt/README_optuna.md

@@ -0,0 +1,105 @@
+# Native Optuna Hyperparameter Optimization
+
+> **Requires Ludwig 0.15 / PR #4090 (data-pipeline-hyperopt-modernization branch).**
+
+Ludwig 0.15 adds a native Optuna executor that runs HPO trials directly without requiring
+Ray Tune. This is the right choice for single-machine HPO: you get AutoSampler, GPSampler
+(Bayesian optimization), TPE, CMA-ES, median / Hyperband pruning, SQLite-backed resumable
+studies, and the optional Optuna dashboard — without the overhead of a Ray cluster.
+
+If you need distributed trials across many GPUs or nodes, keep using the `ray` executor
+(it wraps `OptunaSearch` as its search algorithm). The native executor in this tutorial is
+faster, simpler, and single-process.
+
+## Config
+
+```yaml
+hyperopt:
+  executor:
+    type: optuna
+    num_samples: 50                # how many trials to run
+    sampler: auto                  # auto | gp | tpe | cmaes | random
+    pruner: null                   # null | median | hyperband (optional early stopping)
+    study_name: ludwig_wine_rmse
+    storage: null                  # or sqlite:///wine_hpo.db to persist and resume
+    time_budget_s: 1800
+
+  parameters:
+    trainer.learning_rate:
+      space: loguniform
+      lower: 1e-5
+      upper: 1e-1
+    trainer.batch_size:
+      space: int
+      lower: 32
+      upper: 256
+    combiner.num_fc_layers:
+      space: int
+      lower: 1
+      upper: 4
+    combiner.output_size:
+      space: choice
+      categories: [32, 64, 128, 256]
+
+  output_feature: quality
+  metric: root_mean_squared_error
+  goal: minimize
+  split: validation
+```
+
+### Sampler options
+
+| `sampler` | Description                                              | Rule of thumb                    |
+| --------- | -------------------------------------------------------- | -------------------------------- |
+| `auto`    | Optuna AutoSampler (falls back to TPE on older versions) | Default choice                   |
+| `gp`      | Gaussian-Process Bayesian optimization                   | Continuous spaces, \<100 trials  |
+| `tpe`     | Tree-structured Parzen Estimator                         | Mixed spaces, 50–500 trials      |
+| `cmaes`   | Covariance Matrix Adaptation Evolution Strategy          | Purely-continuous, medium budget |
+| `random`  | Random search (sanity-check baseline)                    | Sanity check                     |
+
+### Persistence and resuming
+
+Set `storage: sqlite:///wine_hpo.db` to persist trials to disk. Re-running with the same
+`study_name` continues the study — failed trials are retried, successful trials are kept.
+
+### Pruning
+
+Set `pruner: median` or `pruner: hyperband` to stop clearly-losing trials early. Requires
+the model code to report intermediate values back (Ludwig's Optuna integration reports the
+validation metric at each epoch so this works out of the box).
+
+## Running
+
+```bash
+pip install 'ludwig[hyperopt]'   # pulls in optuna
+python optuna_executor.py
+```
+
+Expected output (numbers are illustrative):
+
+```
+[Optuna] Best trial:
+  value: 0.6184
+  params:
+    trainer.learning_rate: 0.0032
+    trainer.batch_size:    64
+    combiner.num_fc_layers: 2
+    combiner.output_size:  128
+  completed in: 412.8s
+```
+
+## Files
+
+| File                 | Description                                           |
+| -------------------- | ----------------------------------------------------- |
+| `config_optuna.yaml` | Full hyperopt config using the native Optuna executor |
+| `optuna_executor.py` | Runs `ludwig.hyperopt` with the above config          |
+| `README_optuna.md`   | This file                                             |
+
+## References
+
+- Optuna — Akiba et al., "Optuna: A Next-generation Hyperparameter Optimization Framework",
+  KDD 2019. <https://arxiv.org/abs/1907.10902>
+- AutoSampler — Optuna v4 AutoSampler documentation.
+- Hyperband — Li et al., "Hyperband: A Novel Bandit-Based Approach to Hyperparameter
+  Optimization", JMLR 2018. <https://arxiv.org/abs/1603.06560>

examples/hyperopt/config_optuna.yaml

@@ -0,0 +1,83 @@
+model_type: ecd
+
+input_features:
+  - name: fixed acidity
+    type: number
+    preprocessing:
+      normalization: zscore
+  - name: volatile acidity
+    type: number
+    preprocessing:
+      normalization: zscore
+  - name: citric acid
+    type: number
+    preprocessing:
+      normalization: zscore
+  - name: residual sugar
+    type: number
+    preprocessing:
+      normalization: zscore
+  - name: chlorides
+    type: number
+    preprocessing:
+      normalization: zscore
+  - name: free sulfur dioxide
+    type: number
+    preprocessing:
+      normalization: zscore
+  - name: total sulfur dioxide
+    type: number
+    preprocessing:
+      normalization: zscore
+  - name: density
+    type: number
+    preprocessing:
+      normalization: zscore
+  - name: pH
+    type: number
+    preprocessing:
+      normalization: zscore
+  - name: sulphates
+    type: number
+    preprocessing:
+      normalization: zscore
+  - name: alcohol
+    type: number
+    preprocessing:
+      normalization: zscore
+
+output_features:
+  - name: quality
+    type: binary
+
+trainer:
+  epochs: 20
+
+# NOTE: The Optuna executor requires PR #4090 to be merged, or Ludwig >= 0.14.
+# Install: pip install ludwig optuna
+hyperopt:
+  executor:
+    type: optuna
+    num_samples: 20
+    sampler: auto        # auto selects the best sampler; also: tpe, gp, cmaes, random
+    pruner: hyperband    # stop unpromising trials early (MedianPruner also supported)
+    storage: sqlite:///optuna_results.db  # enables resumability across runs
+  parameters:
+    trainer.learning_rate:
+      space: loguniform
+      lower: 1.0e-5
+      upper: 1.0e-2
+    trainer.batch_size:
+      space: int
+      lower: 16
+      upper: 256
+    trainer.optimizer.type:
+      space: choice
+      categories:
+        - adam
+        - adamw
+        - radam
+        - schedule_free_adamw
+  goal: minimize
+  metric: validation.combined.loss
+  split: validation

examples/hyperopt/optuna_executor.ipynb

@@ -0,0 +1,453 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "7fb27b941602401d91542211134fc71a",
+   "metadata": {},
+   "source": [
+    "# Hyperparameter Optimization with Native Optuna\n",
+    "\n",
+    "This notebook shows how to run Ludwig hyperparameter optimization using the\n",
+    "**native Optuna executor** introduced in PR #4090.\n",
+    "\n",
+    "> **Note:** Requires PR #4090 to be merged, or `pip install ludwig` >= 0.14.\n",
+    "\n",
+    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ludwig-ai/ludwig/blob/main/examples/hyperopt/optuna_executor.ipynb)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "acae54e37e7d407bbb7b55eff062a284",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!pip install ludwig optuna --quiet"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "9a63283cbaf04dbcab1f6479b197f3a8",
+   "metadata": {},
+   "source": [
+    "> **Dependency note:** The `optuna` executor type (`hyperopt.executor.type: optuna`) is\n",
+    "> available from **Ludwig >= 0.14** (merged in PR #4090). Earlier versions only ship the\n",
+    "> Ray Tune executor. To use this notebook with the development branch:\n",
+    ">\n",
+    "> ```bash\n",
+    "> pip install git+https://github.com/ludwig-ai/ludwig.git@data-pipeline-hyperopt-modernization\n",
+    "> ```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8dd0d8092fe74a7c96281538738b07e2",
+   "metadata": {},
+   "source": [
+    "## Dataset\n",
+    "\n",
+    "We use the [UCI Wine Quality dataset](https://archive.ics.uci.edu/ml/datasets/wine+quality).\n",
+    "It contains physicochemical measurements for red and white wines. We combine both files\n",
+    "and create a **binary classification** target: `quality >= 7` → `1` (high quality),\n",
+    "otherwise `0`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "72eea5119410473aa328ad9291626812",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pathlib\n",
+    "import urllib.request\n",
+    "\n",
+    "import pandas as pd\n",
+    "\n",
+    "DATA_DIR = pathlib.Path(\"data\")\n",
+    "DATA_DIR.mkdir(exist_ok=True)\n",
+    "\n",
+    "WHITE_URL = \"https://archive.ics.uci.edu/ml/machine-learning-databases/wine-quality/winequality-white.csv\"\n",
+    "RED_URL = \"https://archive.ics.uci.edu/ml/machine-learning-databases/wine-quality/winequality-red.csv\"\n",
+    "\n",
+    "combined_path = DATA_DIR / \"wine_quality.csv\"\n",
+    "\n",
+    "if not combined_path.exists():\n",
+    "    print(\"Downloading …\")\n",
+    "    urllib.request.urlretrieve(WHITE_URL, DATA_DIR / \"winequality-white.csv\")\n",
+    "    urllib.request.urlretrieve(RED_URL, DATA_DIR / \"winequality-red.csv\")\n",
+    "\n",
+    "    white = pd.read_csv(DATA_DIR / \"winequality-white.csv\", sep=\";\")\n",
+    "    red = pd.read_csv(DATA_DIR / \"winequality-red.csv\", sep=\";\")\n",
+    "    df = pd.concat([white, red], ignore_index=True)\n",
+    "    df[\"quality\"] = (df[\"quality\"] >= 7).astype(int)\n",
+    "    df.to_csv(combined_path, index=False)\n",
+    "else:\n",
+    "    df = pd.read_csv(combined_path)\n",
+    "\n",
+    "print(f\"{len(df)} rows  |  {df['quality'].mean():.1%} positive (quality >= 7)\")\n",
+    "df.head()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8edb47106e1a46a883d545849b8ab81b",
+   "metadata": {},
+   "source": [
+    "## Define search space\n",
+    "\n",
+    "The `hyperopt` section of the Ludwig config specifies:\n",
+    "\n",
+    "- **executor** — which HPO backend to use and how many trials to run\n",
+    "- **parameters** — the search space for each hyperparameter\n",
+    "- **goal / metric** — what to optimise\n",
+    "\n",
+    "The Optuna executor supports the following `space` types:\n",
+    "\n",
+    "| Space | Ludwig key | Description |\n",
+    "|---|---|---|\n",
+    "| Log-uniform float | `loguniform` | Continuous on log scale — ideal for learning rates |\n",
+    "| Uniform float | `float` | Continuous on linear scale — ideal for dropout |\n",
+    "| Integer | `int` | Integer range, linear scale |\n",
+    "| Categorical | `choice` | Discrete set of values |\n",
+    "| Grid | `grid_search` | Exhaustive grid over a list of values |"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "10185d26023b46108eb7d9f57d49d2b3",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Build feature list dynamically from the dataframe\n",
+    "feature_cols = [c for c in df.columns if c != \"quality\"]\n",
+    "\n",
+    "config = {\n",
+    "    \"model_type\": \"ecd\",\n",
+    "    \"input_features\": [\n",
+    "        {\"name\": col, \"type\": \"number\", \"preprocessing\": {\"normalization\": \"zscore\"}} for col in feature_cols\n",
+    "    ],\n",
+    "    \"output_features\": [\n",
+    "        {\"name\": \"quality\", \"type\": \"binary\"},\n",
+    "    ],\n",
+    "    \"trainer\": {\n",
+    "        \"epochs\": 20,\n",
+    "    },\n",
+    "    # NOTE: type: optuna requires Ludwig >= 0.14 (PR #4090)\n",
+    "    \"hyperopt\": {\n",
+    "        \"executor\": {\n",
+    "            \"type\": \"optuna\",\n",
+    "            \"num_samples\": 20,\n",
+    "            \"sampler\": \"auto\",  # auto, tpe, gp, cmaes, random\n",
+    "            \"pruner\": \"hyperband\",  # stop bad trials early\n",
+    "        },\n",
+    "        \"parameters\": {\n",
+    "            \"trainer.learning_rate\": {\n",
+    "                \"space\": \"loguniform\",\n",
+    "                \"lower\": 1e-5,\n",
+    "                \"upper\": 1e-2,\n",
+    "            },\n",
+    "            \"trainer.batch_size\": {\n",
+    "                \"space\": \"int\",\n",
+    "                \"lower\": 16,\n",
+    "                \"upper\": 256,\n",
+    "            },\n",
+    "            \"trainer.optimizer.type\": {\n",
+    "                \"space\": \"choice\",\n",
+    "                \"categories\": [\"adam\", \"adamw\", \"radam\", \"schedule_free_adamw\"],\n",
+    "            },\n",
+    "            \"combiner.dropout\": {\n",
+    "                \"space\": \"float\",\n",
+    "                \"lower\": 0.0,\n",
+    "                \"upper\": 0.5,\n",
+    "            },\n",
+    "        },\n",
+    "        \"goal\": \"minimize\",\n",
+    "        \"metric\": \"validation.combined.loss\",\n",
+    "        \"split\": \"validation\",\n",
+    "    },\n",
+    "}\n",
+    "\n",
+    "import json\n",
+    "\n",
+    "print(json.dumps(config[\"hyperopt\"], indent=2))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8763a12b2bbd4a93a75aff182afb95dc",
+   "metadata": {},
+   "source": [
+    "## Run HPO\n",
+    "\n",
+    "`LudwigModel.hyperopt()` runs the full HPO loop and returns a list of trial results."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "7623eae2785240b9bd12b16a66d81610",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from ludwig.api import LudwigModel\n",
+    "\n",
+    "model = LudwigModel(config=config, logging_level=20)\n",
+    "\n",
+    "hyperopt_results, output_dir, _ = model.hyperopt(\n",
+    "    dataset=str(combined_path),\n",
+    "    output_directory=\"hyperopt_output\",\n",
+    ")\n",
+    "\n",
+    "print(f\"\\nCompleted {len(hyperopt_results)} trials.  Output: {output_dir}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "7cdc8c89c7104fffa095e18ddfef8986",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Show top-5 trials sorted by metric\n",
+    "sorted_results = sorted(hyperopt_results, key=lambda r: r.get(\"metric_score\", float(\"inf\")))\n",
+    "\n",
+    "rows = []\n",
+    "for i, r in enumerate(sorted_results[:5]):\n",
+    "    row = {\"rank\": i + 1, \"loss\": round(r.get(\"metric_score\", float(\"nan\")), 5)}\n",
+    "    row.update(r.get(\"parameters\", {}))\n",
+    "    rows.append(row)\n",
+    "\n",
+    "pd.DataFrame(rows)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b118ea5561624da68c537baed56e602f",
+   "metadata": {},
+   "source": [
+    "## Sampler comparison\n",
+    "\n",
+    "Ludwig's Optuna executor exposes all of Optuna's built-in samplers via the `sampler` key.\n",
+    "\n",
+    "| Sampler | Key | Best for |\n",
+    "|---|---|---|\n",
+    "| **Auto** | `auto` | Default — Optuna selects the best sampler based on search space type |\n",
+    "| **TPE** | `tpe` | General purpose; efficient with < 100 trials; the classic Optuna default |\n",
+    "| **CMA-ES** | `cmaes` | Continuous spaces with many parameters; covariance matrix adaptation |\n",
+    "| **GP (BoTorch)** | `gp` | Sample-efficient Bayesian optimisation; requires `pip install botorch` |\n",
+    "| **Random** | `random` | Baseline; useful for ablations or very large search spaces |\n",
+    "\n",
+    "Change the sampler by editing the executor block:\n",
+    "\n",
+    "```python\n",
+    "\"executor\": {\n",
+    "    \"type\": \"optuna\",\n",
+    "    \"num_samples\": 50,\n",
+    "    \"sampler\": \"tpe\",   # <-- change this\n",
+    "}\n",
+    "```\n",
+    "\n",
+    "For GP, install the optional dependency first:\n",
+    "\n",
+    "```bash\n",
+    "pip install botorch\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "938c804e27f84196a10c8828c723f798",
+   "metadata": {},
+   "source": [
+    "## Resumable HPO with SQLite\n",
+    "\n",
+    "If your HPO run is interrupted (Colab runtime reset, preempted spot instance, etc.) you can\n",
+    "resume from where you left off by pointing Optuna at a **persistent storage** backend.\n",
+    "\n",
+    "Add a `storage` key to the executor:\n",
+    "\n",
+    "```python\n",
+    "\"executor\": {\n",
+    "    \"type\": \"optuna\",\n",
+    "    \"num_samples\": 50,\n",
+    "    \"sampler\": \"auto\",\n",
+    "    \"storage\": \"sqlite:///optuna_results.db\",  # <-- persist to disk\n",
+    "}\n",
+    "```\n",
+    "\n",
+    "Re-running `model.hyperopt()` with the same storage path will **continue the existing\n",
+    "study** rather than starting a new one. Optuna automatically detects how many trials\n",
+    "have already been completed and runs only the remaining ones.\n",
+    "\n",
+    "For distributed or cloud setups you can also use a PostgreSQL URL:\n",
+    "\n",
+    "```python\n",
+    "\"storage\": \"postgresql://user:pass@host/dbname\"\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "504fb2a444614c0babb325280ed9130a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Example: run with SQLite storage for resumability\n",
+    "import copy\n",
+    "\n",
+    "config_resumable = copy.deepcopy(config)\n",
+    "config_resumable[\"hyperopt\"][\"executor\"][\"storage\"] = \"sqlite:///optuna_results.db\"\n",
+    "config_resumable[\"hyperopt\"][\"executor\"][\"num_samples\"] = 10  # fewer trials for demo\n",
+    "\n",
+    "print(\"Executor config:\")\n",
+    "print(json.dumps(config_resumable[\"hyperopt\"][\"executor\"], indent=2))\n",
+    "print(\"\\nRe-running with storage enabled — existing trials will be reused.\")\n",
+    "\n",
+    "model2 = LudwigModel(config=config_resumable, logging_level=20)\n",
+    "results2, _, _ = model2.hyperopt(\n",
+    "    dataset=str(combined_path),\n",
+    "    output_directory=\"hyperopt_output_resumable\",\n",
+    ")\n",
+    "print(f\"Done. {len(results2)} trials.\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "59bbdb311c014d738909a11f9e486628",
+   "metadata": {},
+   "source": [
+    "## Pruner: stop bad trials early\n",
+    "\n",
+    "A **pruner** monitors intermediate results reported during training and stops trials that\n",
+    "are unlikely to beat the current best. This can dramatically reduce total compute when\n",
+    "combined with epoch-level reporting.\n",
+    "\n",
+    "Ludwig's Optuna executor supports:\n",
+    "\n",
+    "| Pruner | Key | Description |\n",
+    "|---|---|---|\n",
+    "| **Hyperband** | `hyperband` | Successive halving over training steps; efficient for deep learning |\n",
+    "| **Median** | `median` | Stops trials below the median performance at a given step |\n",
+    "| **None** | *(omit key)* | No pruning; every trial runs to completion |\n",
+    "\n",
+    "```python\n",
+    "\"executor\": {\n",
+    "    \"type\": \"optuna\",\n",
+    "    \"num_samples\": 50,\n",
+    "    \"sampler\": \"auto\",\n",
+    "    \"pruner\": \"hyperband\",  # <-- add this\n",
+    "}\n",
+    "```\n",
+    "\n",
+    "Hyperband is the recommended default for neural network HPO. It requires at least\n",
+    "`min_resource` epochs (default 1) to have completed before making pruning decisions,\n",
+    "so short-running models (< 5 epochs) may see limited benefit."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b43b363d81ae4b689946ece5c682cd59",
+   "metadata": {},
+   "source": [
+    "## Results\n",
+    "\n",
+    "The cells below plot a **parallel coordinates** chart — each line is one trial,\n",
+    "colour-coded by the validation loss. Narrow bundles indicate which regions of\n",
+    "the search space consistently produce good results."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8a65eabff63a45729fe45fb5ade58bdc",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Build a dataframe of all trial results\n",
+    "records = []\n",
+    "for r in hyperopt_results:\n",
+    "    row = {\"loss\": r.get(\"metric_score\", float(\"nan\"))}\n",
+    "    row.update(r.get(\"parameters\", {}))\n",
+    "    records.append(row)\n",
+    "\n",
+    "results_df = pd.DataFrame(records)\n",
+    "print(f\"{len(results_df)} trials\")\n",
+    "results_df.describe()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "c3933fab20d04ec698c2621248eb3be0",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import plotly.express as px\n",
+    "\n",
+    "# Map categorical optimizer to numeric for colour scale\n",
+    "opt_map = {v: i for i, v in enumerate(results_df[\"trainer.optimizer.type\"].unique())}\n",
+    "results_df[\"optimizer_idx\"] = results_df[\"trainer.optimizer.type\"].map(opt_map)\n",
+    "\n",
+    "dims = [\n",
+    "    dict(label=\"learning_rate\", values=results_df[\"trainer.learning_rate\"], type=\"log\"),\n",
+    "    dict(label=\"batch_size\", values=results_df[\"trainer.batch_size\"]),\n",
+    "    dict(\n",
+    "        label=\"optimizer\",\n",
+    "        values=results_df[\"optimizer_idx\"],\n",
+    "        tickvals=list(opt_map.values()),\n",
+    "        ticktext=list(opt_map.keys()),\n",
+    "    ),\n",
+    "    dict(label=\"dropout\", values=results_df[\"combiner.dropout\"]),\n",
+    "    dict(label=\"val loss\", values=results_df[\"loss\"]),\n",
+    "]\n",
+    "\n",
+    "fig = px.parallel_coordinates(\n",
+    "    results_df,\n",
+    "    dimensions=[\"trainer.learning_rate\", \"trainer.batch_size\", \"optimizer_idx\", \"combiner.dropout\", \"loss\"],\n",
+    "    color=\"loss\",\n",
+    "    color_continuous_scale=px.colors.sequential.Viridis_r,\n",
+    "    labels={\n",
+    "        \"trainer.learning_rate\": \"learning rate\",\n",
+    "        \"trainer.batch_size\": \"batch size\",\n",
+    "        \"optimizer_idx\": \"optimizer\",\n",
+    "        \"combiner.dropout\": \"dropout\",\n",
+    "        \"loss\": \"val loss\",\n",
+    "    },\n",
+    "    title=\"HPO trials — parallel coordinates (lower loss is better)\",\n",
+    ")\n",
+    "fig.show()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "4dd4641cc4064e0191573fe9c69df29b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Print best configuration\n",
+    "best = sorted_results[0]\n",
+    "print(f\"Best validation loss : {best['metric_score']:.5f}\")\n",
+    "print(\"\\nBest hyperparameters:\")\n",
+    "for k, v in best[\"parameters\"].items():\n",
+    "    print(f\"  {k:35s} = {v}\")"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "name": "python",
+   "version": "3.12.0"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

examples/hyperopt/optuna_executor.py

@@ -0,0 +1,145 @@
+"""
+Hyperparameter Optimization with Native Optuna Executor
+========================================================
+
+NOTE: Requires PR #4090 to be merged, or Ludwig >= 0.14.
+      Install dependencies: pip install ludwig optuna
+
+Usage:
+    python optuna_executor.py
+
+The script downloads the UCI Wine Quality dataset, binarises the target
+(quality >= 7), and runs Ludwig HPO using the native Optuna executor.
+Results are persisted in `optuna_results.db` so interrupted runs can
+be resumed by simply re-running the script.
+"""
+
+# Colab: !pip install ludwig optuna --quiet
+
+import pathlib
+import urllib.request
+
+import pandas as pd
+
+# ---------------------------------------------------------------------------
+# 1. Download dataset
+# ---------------------------------------------------------------------------
+DATA_DIR = pathlib.Path("data")
+DATA_DIR.mkdir(exist_ok=True)
+
+WHITE_URL = "https://archive.ics.uci.edu/ml/machine-learning-databases/wine-quality/winequality-white.csv"
+RED_URL = "https://archive.ics.uci.edu/ml/machine-learning-databases/wine-quality/winequality-red.csv"
+
+white_path = DATA_DIR / "winequality-white.csv"
+red_path = DATA_DIR / "winequality-red.csv"
+combined_path = DATA_DIR / "wine_quality.csv"
+
+if not combined_path.exists():
+    print("Downloading Wine Quality dataset from UCI …")
+    urllib.request.urlretrieve(WHITE_URL, white_path)
+    urllib.request.urlretrieve(RED_URL, red_path)
+
+    white = pd.read_csv(white_path, sep=";")
+    red = pd.read_csv(red_path, sep=";")
+    df = pd.concat([white, red], ignore_index=True)
+
+    # Binary target: 1 if quality >= 7, else 0
+    df["quality"] = (df["quality"] >= 7).astype(int)
+
+    df.to_csv(combined_path, index=False)
+    print(f"Dataset saved to {combined_path} ({len(df)} rows)")
+else:
+    print(f"Dataset already present at {combined_path}")
+    df = pd.read_csv(combined_path)
+    print(f"  {len(df)} rows, class balance: {df['quality'].mean():.1%} positive")
+
+# ---------------------------------------------------------------------------
+# 2. Ludwig config
+# ---------------------------------------------------------------------------
+config = {
+    "model_type": "ecd",
+    "input_features": [
+        {"name": col, "type": "number", "preprocessing": {"normalization": "zscore"}}
+        for col in df.columns
+        if col != "quality"
+    ],
+    "output_features": [
+        {"name": "quality", "type": "binary"},
+    ],
+    "trainer": {
+        "epochs": 20,
+    },
+    # NOTE: Optuna executor is available from Ludwig >= 0.14 (PR #4090).
+    "hyperopt": {
+        "executor": {
+            "type": "optuna",
+            "num_samples": 20,
+            # 'auto' lets Optuna pick the best sampler given the search space.
+            # Alternatives: 'tpe', 'gp', 'cmaes', 'random'
+            "sampler": "auto",
+            # Hyperband pruner stops unpromising trials early.
+            "pruner": "hyperband",
+            # SQLite storage makes runs resumable: re-run the script and
+            # Optuna will continue from where it left off.
+            "storage": "sqlite:///optuna_results.db",
+        },
+        "parameters": {
+            "trainer.learning_rate": {
+                "space": "loguniform",
+                "lower": 1e-5,
+                "upper": 1e-2,
+            },
+            "trainer.batch_size": {
+                "space": "int",
+                "lower": 16,
+                "upper": 256,
+            },
+            "trainer.optimizer.type": {
+                "space": "choice",
+                "categories": ["adam", "adamw", "radam", "schedule_free_adamw"],
+            },
+            "combiner.dropout": {
+                "space": "float",
+                "lower": 0.0,
+                "upper": 0.5,
+            },
+        },
+        "goal": "minimize",
+        "metric": "validation.combined.loss",
+        "split": "validation",
+    },
+}
+
+# ---------------------------------------------------------------------------
+# 3. Run HPO
+# ---------------------------------------------------------------------------
+try:
+    from ludwig.api import LudwigModel
+except ImportError:
+    raise SystemExit("Ludwig is not installed. Run: pip install ludwig optuna")
+
+print("\nStarting hyperparameter optimisation with Optuna …")
+print("Results are persisted in optuna_results.db — re-run to resume.\n")
+
+model = LudwigModel(config=config, logging_level=20)  # INFO
+
+hyperopt_results, _, _ = model.hyperopt(
+    dataset=str(combined_path),
+    output_directory="hyperopt_output",
+)
+
+# ---------------------------------------------------------------------------
+# 4. Report results
+# ---------------------------------------------------------------------------
+print("\n" + "=" * 60)
+print("HPO complete")
+print("=" * 60)
+
+if hyperopt_results:
+    best = min(hyperopt_results, key=lambda r: r.get("metric_score", float("inf")))
+    print(f"\nBest metric (validation.combined.loss): {best.get('metric_score', 'n/a'):.4f}")
+    print("\nBest hyperparameters:")
+    for param, value in best.get("parameters", {}).items():
+        print(f"  {param}: {value}")
+else:
+    print("No results returned — check logs above for errors.")

examples/image_encoders/README.md

@@ -0,0 +1,78 @@
+# Pretrained Image Encoders: CLIP, DINOv2, and SigLIP
+
+[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ludwig-ai/ludwig/blob/main/examples/image_encoders/image_encoders.ipynb)
+
+## Overview
+
+Pretrained image encoders are neural networks trained on large datasets (ImageNet-21k, LAION-5B, or proprietary corpora) whose learned weights can be directly transferred to new tasks. Instead of training a convolutional network from scratch on your small dataset, you can use a pretrained encoder as a frozen feature extractor and only train a lightweight classification head on top—this is called **linear probing**.
+
+### Why pretrained encoders matter for few-shot learning
+
+When you have limited labeled data (e.g., 5–100 examples per class), training from scratch typically leads to overfitting. Pretrained encoders solve this by:
+
+- Providing rich, general-purpose visual features learned from millions of images
+- Allowing the model to converge in far fewer epochs
+- Requiring only a small head to be trained, which needs very little data
+
+Ludwig supports three HuggingFace-backed pretrained image encoders alongside the traditional `stacked_cnn` approach.
+
+## Encoder comparison
+
+| Encoder       | Pretrained | Trainable by default | Best for                                                                    |
+| ------------- | ---------- | -------------------- | --------------------------------------------------------------------------- |
+| `stacked_cnn` | No         | Yes                  | Full control, small images, custom architectures                            |
+| `dinov2`      | Yes        | Yes                  | General image classification, dense prediction, linear probing              |
+| `clip`        | Yes        | Yes                  | Image-text tasks, zero-shot classification, multimodal fusion               |
+| `siglip`      | Yes        | Yes                  | CLIP-like tasks with better scaling, Google's improved contrastive training |
+
+All three pretrained encoders (`dinov2`, `clip`, `siglip`) support:
+
+- `use_pretrained: true` — load weights from HuggingFace Hub
+- `trainable: false` — freeze the encoder for fast linear probing
+- `trainable: true` — fine-tune the full encoder end-to-end
+
+## Quick start
+
+Install Ludwig with vision support:
+
+```bash
+pip install ludwig[vision]
+```
+
+Train with a pretrained DINOv2 encoder (linear probe — fast, works well with limited data):
+
+```bash
+ludwig train \
+  --config examples/image_encoders/config_dinov2_linear_probe.yaml \
+  --dataset my_images.csv
+```
+
+Your CSV needs two columns: `image_path` (absolute or relative paths to image files) and `label` (the class name).
+
+## Available configs
+
+| Config file                       | Description                                   |
+| --------------------------------- | --------------------------------------------- |
+| `config_stacked_cnn.yaml`         | CNN trained from scratch (20 epochs)          |
+| `config_dinov2_linear_probe.yaml` | DINOv2 frozen backbone, head only (10 epochs) |
+| `config_dinov2_finetuned.yaml`    | DINOv2 full fine-tune (5 epochs, lower LR)    |
+| `config_clip.yaml`                | CLIP frozen backbone (10 epochs)              |
+| `config_siglip.yaml`              | SigLIP frozen backbone (10 epochs)            |
+
+## Running all configs and comparing results
+
+```bash
+python examples/image_encoders/compare_encoders.py --dataset my_images.csv
+```
+
+## Full walkthrough
+
+See the [notebook](image_encoders.ipynb) for a complete step-by-step example using the `beans` plant disease dataset (3 classes, ~1000 images) from HuggingFace Datasets, including a few-shot experiment with only 15 training examples.
+
+## Hardware requirements
+
+- `stacked_cnn`: CPU or GPU
+- `dinov2` (linear probe): GPU recommended, runs on CPU for small datasets
+- `dinov2` (fine-tune), `clip`, `siglip`: GPU required (T4 or better)
+
+The linear probe is especially well-suited for Google Colab free tier (T4 GPU).

examples/image_encoders/compare_encoders.py

@@ -0,0 +1,183 @@
+"""Compare pretrained image encoders: stacked_cnn, DINOv2, CLIP, and SigLIP.
+
+Runs all encoder configs on a dataset and prints a comparison table of
+accuracy, training time, and approximate GPU memory usage.
+
+Usage:
+    python compare_encoders.py --dataset /path/to/data.csv
+
+The dataset CSV must have columns: image_path, label
+
+Example (using the beans dataset — see the notebook for download instructions):
+    python compare_encoders.py --dataset /tmp/beans/train.csv \
+        --val_dataset /tmp/beans/validation.csv \
+        --test_dataset /tmp/beans/test.csv
+"""
+
+import argparse
+import os
+import time
+from pathlib import Path
+
+SCRIPT_DIR = Path(__file__).parent
+
+
+def get_gpu_memory_mb() -> float:
+    """Return current GPU memory usage in MB, or 0 if no GPU available."""
+    try:
+        import torch
+
+        if torch.cuda.is_available():
+            return torch.cuda.max_memory_allocated() / 1024**2
+    except ImportError:
+        pass
+    return 0.0
+
+
+def reset_gpu_memory_stats() -> None:
+    try:
+        import torch
+
+        if torch.cuda.is_available():
+            torch.cuda.reset_peak_memory_stats()
+    except ImportError:
+        pass
+
+
+def run_experiment(
+    config_path: str,
+    dataset: str,
+    val_dataset: str | None,
+    test_dataset: str | None,
+    output_dir: str,
+) -> dict:
+    """Train and evaluate one config.
+
+    Returns a dict with result metrics.
+    """
+    from ludwig.api import LudwigModel
+
+    reset_gpu_memory_stats()
+    start = time.time()
+
+    model = LudwigModel(config=config_path, logging_level=30)  # WARNING level
+
+    train_kwargs = dict(dataset=dataset, output_directory=output_dir, skip_save_processed_input=True)
+    if val_dataset:
+        train_kwargs["validation_set"] = val_dataset
+
+    _, _, output_directory = model.train(**train_kwargs)
+    train_time = time.time() - start
+    peak_mem = get_gpu_memory_mb()
+
+    # Evaluate on test set
+    eval_dataset = test_dataset or dataset
+    eval_stats, _, _ = model.evaluate(dataset=eval_dataset, collect_overall_stats=True)
+
+    accuracy = 0.0
+    if "label" in eval_stats:
+        accuracy = eval_stats["label"].get("accuracy", 0.0)
+
+    return {
+        "train_time_s": train_time,
+        "peak_gpu_mb": peak_mem,
+        "accuracy": accuracy,
+        "output_directory": output_directory,
+    }
+
+
+CONFIGS = [
+    ("stacked_cnn", "config_stacked_cnn.yaml"),
+    ("dinov2_linear_probe", "config_dinov2_linear_probe.yaml"),
+    ("dinov2_finetuned", "config_dinov2_finetuned.yaml"),
+    ("clip", "config_clip.yaml"),
+    ("siglip", "config_siglip.yaml"),
+]
+
+
+def print_table(results: list[dict]) -> None:
+    header = f"{'Encoder':<25} {'Accuracy':>10} {'Train time':>12} {'Peak GPU (MB)':>15}"
+    sep = "-" * len(header)
+    print("\n" + sep)
+    print(header)
+    print(sep)
+    for r in results:
+        name = r["name"]
+        acc = r.get("accuracy", float("nan"))
+        t = r.get("train_time_s", float("nan"))
+        mem = r.get("peak_gpu_mb", 0.0)
+        mem_str = f"{mem:>15.0f}" if mem > 0 else f"{'N/A':>15}"
+        print(f"{name:<25} {acc:>10.4f} {t:>11.1f}s {mem_str}")
+    print(sep + "\n")
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description="Compare Ludwig image encoder configs.")
+    parser.add_argument("--dataset", required=True, help="Path to training CSV (image_path, label columns).")
+    parser.add_argument("--val_dataset", default=None, help="Path to validation CSV.")
+    parser.add_argument("--test_dataset", default=None, help="Path to test CSV (used for evaluation).")
+    parser.add_argument(
+        "--output_dir",
+        default="/tmp/image_encoder_results",
+        help="Base output directory for Ludwig experiment results.",
+    )
+    parser.add_argument(
+        "--encoders",
+        nargs="+",
+        default=None,
+        help="Subset of encoders to run (e.g. --encoders stacked_cnn dinov2_linear_probe). Defaults to all encoders.",
+    )
+    args = parser.parse_args()
+
+    configs_to_run = CONFIGS
+    if args.encoders:
+        valid = {name for name, _ in CONFIGS}
+        for e in args.encoders:
+            if e not in valid:
+                parser.error(f"Unknown encoder '{e}'. Valid: {sorted(valid)}")
+        configs_to_run = [(name, cfg) for name, cfg in CONFIGS if name in args.encoders]
+
+    results = []
+    for name, config_file in configs_to_run:
+        config_path = str(SCRIPT_DIR / config_file)
+        if not os.path.exists(config_path):
+            print(f"[SKIP] Config not found: {config_path}")
+            continue
+
+        output_dir = os.path.join(args.output_dir, name)
+        os.makedirs(output_dir, exist_ok=True)
+
+        print(f"\n{'=' * 60}")
+        print(f"Running: {name}")
+        print(f"Config:  {config_path}")
+        print(f"{'=' * 60}")
+
+        try:
+            metrics = run_experiment(
+                config_path=config_path,
+                dataset=args.dataset,
+                val_dataset=args.val_dataset,
+                test_dataset=args.test_dataset,
+                output_dir=output_dir,
+            )
+            results.append({"name": name, **metrics})
+            print(
+                f"Done: accuracy={metrics['accuracy']:.4f}, "
+                f"time={metrics['train_time_s']:.1f}s, "
+                f"peak_gpu={metrics['peak_gpu_mb']:.0f}MB"
+            )
+        except Exception as exc:
+            print(f"[ERROR] {name} failed: {exc}")
+            results.append({"name": name, "error": str(exc)})
+
+    print_table([r for r in results if "error" not in r])
+
+    failed = [r for r in results if "error" in r]
+    if failed:
+        print("Failed experiments:")
+        for r in failed:
+            print(f"  {r['name']}: {r['error']}")
+
+
+if __name__ == "__main__":
+    main()

examples/image_encoders/config_clip.yaml

@@ -0,0 +1,14 @@
+model_type: ecd
+input_features:
+  - name: image_path
+    type: image
+    encoder:
+      type: clip
+      use_pretrained: true
+      trainable: false
+output_features:
+  - name: label
+    type: category
+trainer:
+  epochs: 10
+  learning_rate: 0.001

examples/image_encoders/config_dinov2_finetuned.yaml

@@ -0,0 +1,14 @@
+model_type: ecd
+input_features:
+  - name: image_path
+    type: image
+    encoder:
+      type: dinov2
+      use_pretrained: true
+      trainable: true
+output_features:
+  - name: label
+    type: category
+trainer:
+  epochs: 5
+  learning_rate: 0.0001

examples/image_encoders/config_dinov2_linear_probe.yaml

@@ -0,0 +1,14 @@
+model_type: ecd
+input_features:
+  - name: image_path
+    type: image
+    encoder:
+      type: dinov2
+      use_pretrained: true
+      trainable: false
+output_features:
+  - name: label
+    type: category
+trainer:
+  epochs: 10
+  learning_rate: 0.001

examples/image_encoders/config_siglip.yaml

@@ -0,0 +1,14 @@
+model_type: ecd
+input_features:
+  - name: image_path
+    type: image
+    encoder:
+      type: siglip
+      use_pretrained: true
+      trainable: false
+output_features:
+  - name: label
+    type: category
+trainer:
+  epochs: 10
+  learning_rate: 0.001

examples/image_encoders/config_stacked_cnn.yaml

@@ -0,0 +1,13 @@
+model_type: ecd
+input_features:
+  - name: image_path
+    type: image
+    encoder:
+      type: stacked_cnn
+      use_pretrained: false
+output_features:
+  - name: label
+    type: category
+trainer:
+  epochs: 20
+  learning_rate: 0.001

examples/image_encoders/image_encoders.ipynb

@@ -0,0 +1,741 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "a1b2c3d4",
+   "metadata": {},
+   "source": [
+    "# Pretrained Image Encoders: CLIP, DINOv2, and SigLIP\n",
+    "\n",
+    "This notebook compares four approaches to image classification in Ludwig:\n",
+    "\n",
+    "1. **stacked_cnn** — a convolutional network trained from scratch\n",
+    "2. **DINOv2 linear probe** — pretrained DINOv2 backbone, frozen; only the head is trained\n",
+    "3. **DINOv2 fine-tuned** — pretrained DINOv2 backbone, full fine-tuning\n",
+    "4. **CLIP** — pretrained CLIP vision encoder, frozen\n",
+    "5. **SigLIP** — pretrained SigLIP vision encoder, frozen\n",
+    "\n",
+    "We use the [`beans`](https://huggingface.co/datasets/beans) dataset: a plant disease dataset with 3 classes (~1000 images total), which makes it a realistic small-data scenario where pretrained encoders shine.\n",
+    "\n",
+    "**Runtime**: GPU is required for DINOv2/CLIP/SigLIP. In Colab, go to **Runtime → Change runtime type → T4 GPU**."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b2c3d4e5",
+   "metadata": {},
+   "source": [
+    "## Setup"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "c3d4e5f6",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!pip install ludwig[vision] datasets --quiet"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d4e5f6a7",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import time\n",
+    "import warnings\n",
+    "\n",
+    "import torch\n",
+    "\n",
+    "warnings.filterwarnings(\"ignore\")\n",
+    "\n",
+    "print(f\"PyTorch version: {torch.__version__}\")\n",
+    "print(f\"CUDA available: {torch.cuda.is_available()}\")\n",
+    "if torch.cuda.is_available():\n",
+    "    print(f\"GPU: {torch.cuda.get_device_name(0)}\")\n",
+    "    print(f\"GPU memory: {torch.cuda.get_device_properties(0).total_memory / 1024**3:.1f} GB\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e5f6a7b8",
+   "metadata": {},
+   "source": [
+    "## Dataset\n",
+    "\n",
+    "The `beans` dataset contains RGB images of bean leaves classified into three categories:\n",
+    "- `angular_leaf_spot` — fungal disease\n",
+    "- `bean_rust` — rust disease  \n",
+    "- `healthy` — healthy leaf\n",
+    "\n",
+    "It has ~1034 training images, 133 validation images, and 128 test images. This makes it a realistic small-data scenario."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f6a7b8c9",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import csv\n",
+    "from pathlib import Path\n",
+    "\n",
+    "from datasets import load_dataset\n",
+    "\n",
+    "# Download the beans dataset from HuggingFace\n",
+    "ds = load_dataset(\"beans\")\n",
+    "print(ds)\n",
+    "\n",
+    "# Save images to disk and create CSV files\n",
+    "DATA_DIR = Path(\"/tmp/beans\")\n",
+    "DATA_DIR.mkdir(parents=True, exist_ok=True)\n",
+    "\n",
+    "label_names = ds[\"train\"].features[\"labels\"].names\n",
+    "print(f\"Classes: {label_names}\")\n",
+    "\n",
+    "\n",
+    "def save_split(split_name: str) -> str:\n",
+    "    \"\"\"Save images to disk and return path to a CSV with image_path and label columns.\"\"\"\n",
+    "    split_dir = DATA_DIR / split_name\n",
+    "    split_dir.mkdir(parents=True, exist_ok=True)\n",
+    "    csv_path = DATA_DIR / f\"{split_name}.csv\"\n",
+    "\n",
+    "    rows = []\n",
+    "    for i, example in enumerate(ds[split_name]):\n",
+    "        img = example[\"image\"]\n",
+    "        label_idx = example[\"labels\"]\n",
+    "        label = label_names[label_idx]\n",
+    "\n",
+    "        img_path = split_dir / f\"{i:04d}_{label}.jpg\"\n",
+    "        if not img_path.exists():\n",
+    "            img.save(str(img_path))\n",
+    "\n",
+    "        rows.append({\"image_path\": str(img_path), \"label\": label})\n",
+    "\n",
+    "    with open(csv_path, \"w\", newline=\"\") as f:\n",
+    "        writer = csv.DictWriter(f, fieldnames=[\"image_path\", \"label\"])\n",
+    "        writer.writeheader()\n",
+    "        writer.writerows(rows)\n",
+    "\n",
+    "    print(f\"Saved {len(rows)} examples to {csv_path}\")\n",
+    "    return str(csv_path)\n",
+    "\n",
+    "\n",
+    "train_csv = save_split(\"train\")\n",
+    "val_csv = save_split(\"validation\")\n",
+    "test_csv = save_split(\"test\")\n",
+    "\n",
+    "print(f\"\\nTrain: {train_csv}\")\n",
+    "print(f\"Val:   {val_csv}\")\n",
+    "print(f\"Test:  {test_csv}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a7b8c9d0",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Preview a few examples\n",
+    "import pandas as pd\n",
+    "\n",
+    "train_df = pd.read_csv(train_csv)\n",
+    "print(train_df.head())\n",
+    "print(f\"\\nClass distribution:\\n{train_df['label'].value_counts()}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b8c9d0e1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Visualize a few images\n",
+    "import matplotlib.pyplot as plt\n",
+    "from PIL import Image\n",
+    "\n",
+    "fig, axes = plt.subplots(1, 3, figsize=(12, 4))\n",
+    "for i, (label, group) in enumerate(train_df.groupby(\"label\")):\n",
+    "    sample = group.sample(1).iloc[0]\n",
+    "    img = Image.open(sample[\"image_path\"])\n",
+    "    axes[i].imshow(img)\n",
+    "    axes[i].set_title(label)\n",
+    "    axes[i].axis(\"off\")\n",
+    "plt.suptitle(\"Beans dataset — one example per class\", fontsize=13)\n",
+    "plt.tight_layout()\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c9d0e1f2",
+   "metadata": {},
+   "source": [
+    "## Helper: timing and memory tracking"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d0e1f2a3",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import gc\n",
+    "\n",
+    "results = {}  # Will accumulate {encoder_name: {accuracy, train_time_s, peak_gpu_mb}}\n",
+    "\n",
+    "\n",
+    "def reset_gpu():\n",
+    "    gc.collect()\n",
+    "    if torch.cuda.is_available():\n",
+    "        torch.cuda.empty_cache()\n",
+    "        torch.cuda.reset_peak_memory_stats()\n",
+    "\n",
+    "\n",
+    "def peak_gpu_mb() -> float:\n",
+    "    if torch.cuda.is_available():\n",
+    "        return torch.cuda.max_memory_allocated() / 1024**2\n",
+    "    return 0.0\n",
+    "\n",
+    "\n",
+    "def evaluate_model(model, test_csv: str) -> float:\n",
+    "    \"\"\"Return accuracy on the test set.\"\"\"\n",
+    "    eval_stats, _, _ = model.evaluate(dataset=test_csv, collect_overall_stats=True)\n",
+    "    return eval_stats.get(\"label\", {}).get(\"accuracy\", float(\"nan\"))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e1f2a3b4",
+   "metadata": {},
+   "source": [
+    "## Approach 1: Stacked CNN (trained from scratch)\n",
+    "\n",
+    "The `stacked_cnn` encoder is Ludwig's default: a stack of 2D convolutional layers followed by fully connected layers. There are no pretrained weights — the model learns everything from the 1034 training images.\n",
+    "\n",
+    "This is the baseline. Expect lower accuracy and longer training time compared to pretrained encoders on this small dataset."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f2a3b4c5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from ludwig.api import LudwigModel\n",
+    "\n",
+    "config_stacked_cnn = {\n",
+    "    \"model_type\": \"ecd\",\n",
+    "    \"input_features\": [\n",
+    "        {\n",
+    "            \"name\": \"image_path\",\n",
+    "            \"type\": \"image\",\n",
+    "            \"encoder\": {\n",
+    "                \"type\": \"stacked_cnn\",\n",
+    "                \"use_pretrained\": False,\n",
+    "            },\n",
+    "        }\n",
+    "    ],\n",
+    "    \"output_features\": [{\"name\": \"label\", \"type\": \"category\"}],\n",
+    "    \"trainer\": {\"epochs\": 20, \"learning_rate\": 0.001},\n",
+    "}\n",
+    "\n",
+    "reset_gpu()\n",
+    "t0 = time.time()\n",
+    "\n",
+    "model_cnn = LudwigModel(config=config_stacked_cnn, logging_level=30)\n",
+    "_, _, output_dir = model_cnn.train(\n",
+    "    dataset=train_csv,\n",
+    "    validation_set=val_csv,\n",
+    "    output_directory=\"/tmp/results/stacked_cnn\",\n",
+    "    skip_save_processed_input=True,\n",
+    ")\n",
+    "\n",
+    "train_time = time.time() - t0\n",
+    "gpu_mem = peak_gpu_mb()\n",
+    "accuracy = evaluate_model(model_cnn, test_csv)\n",
+    "\n",
+    "results[\"stacked_cnn\"] = {\"accuracy\": accuracy, \"train_time_s\": train_time, \"peak_gpu_mb\": gpu_mem}\n",
+    "print(f\"stacked_cnn — accuracy: {accuracy:.4f}, time: {train_time:.1f}s, peak GPU: {gpu_mem:.0f}MB\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a3b4c5d6",
+   "metadata": {},
+   "source": [
+    "## Approach 2: DINOv2 linear probe\n",
+    "\n",
+    "DINOv2 (Oquab et al., TMLR 2024) is a self-supervised vision transformer from Meta trained on 142M images. It produces rich general-purpose features without requiring image-text pairs.\n",
+    "\n",
+    "In **linear probe** mode (`trainable: false`), the DINOv2 backbone is frozen — only the Ludwig output head (a small linear layer) is trained. This is:\n",
+    "- **Fast**: no backprop through the large backbone\n",
+    "- **Data-efficient**: works well with very few labeled examples\n",
+    "- **Memory-efficient**: no gradients stored for the backbone\n",
+    "\n",
+    "Default model: `facebook/dinov2-base` (86M parameters, 768-dim output)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b4c5d6e7",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "config_dinov2_probe = {\n",
+    "    \"model_type\": \"ecd\",\n",
+    "    \"input_features\": [\n",
+    "        {\n",
+    "            \"name\": \"image_path\",\n",
+    "            \"type\": \"image\",\n",
+    "            \"encoder\": {\n",
+    "                \"type\": \"dinov2\",\n",
+    "                \"use_pretrained\": True,\n",
+    "                \"trainable\": False,  # freeze the backbone\n",
+    "            },\n",
+    "        }\n",
+    "    ],\n",
+    "    \"output_features\": [{\"name\": \"label\", \"type\": \"category\"}],\n",
+    "    \"trainer\": {\"epochs\": 10, \"learning_rate\": 0.001},\n",
+    "}\n",
+    "\n",
+    "reset_gpu()\n",
+    "t0 = time.time()\n",
+    "\n",
+    "model_dinov2_probe = LudwigModel(config=config_dinov2_probe, logging_level=30)\n",
+    "_, _, output_dir = model_dinov2_probe.train(\n",
+    "    dataset=train_csv,\n",
+    "    validation_set=val_csv,\n",
+    "    output_directory=\"/tmp/results/dinov2_linear_probe\",\n",
+    "    skip_save_processed_input=True,\n",
+    ")\n",
+    "\n",
+    "train_time = time.time() - t0\n",
+    "gpu_mem = peak_gpu_mb()\n",
+    "accuracy = evaluate_model(model_dinov2_probe, test_csv)\n",
+    "\n",
+    "results[\"dinov2_linear_probe\"] = {\"accuracy\": accuracy, \"train_time_s\": train_time, \"peak_gpu_mb\": gpu_mem}\n",
+    "print(f\"DINOv2 linear probe — accuracy: {accuracy:.4f}, time: {train_time:.1f}s, peak GPU: {gpu_mem:.0f}MB\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c5d6e7f8",
+   "metadata": {},
+   "source": [
+    "## Approach 3: DINOv2 fine-tuned\n",
+    "\n",
+    "In **fine-tuning** mode (`trainable: true`), gradients flow through the entire DINOv2 backbone. This can improve accuracy over linear probing but requires:\n",
+    "- More GPU memory (stores activations for backprop through 86M parameters)\n",
+    "- A lower learning rate (to avoid destroying pretrained features)\n",
+    "- Fewer epochs (the model has a strong starting point)\n",
+    "\n",
+    "Use fine-tuning when you have enough data and GPU memory and want maximum accuracy."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d6e7f8a9",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "config_dinov2_finetune = {\n",
+    "    \"model_type\": \"ecd\",\n",
+    "    \"input_features\": [\n",
+    "        {\n",
+    "            \"name\": \"image_path\",\n",
+    "            \"type\": \"image\",\n",
+    "            \"encoder\": {\n",
+    "                \"type\": \"dinov2\",\n",
+    "                \"use_pretrained\": True,\n",
+    "                \"trainable\": True,  # full fine-tuning\n",
+    "            },\n",
+    "        }\n",
+    "    ],\n",
+    "    \"output_features\": [{\"name\": \"label\", \"type\": \"category\"}],\n",
+    "    \"trainer\": {\n",
+    "        \"epochs\": 5,\n",
+    "        \"learning_rate\": 0.0001,  # lower LR for fine-tuning\n",
+    "    },\n",
+    "}\n",
+    "\n",
+    "reset_gpu()\n",
+    "t0 = time.time()\n",
+    "\n",
+    "model_dinov2_ft = LudwigModel(config=config_dinov2_finetune, logging_level=30)\n",
+    "_, _, output_dir = model_dinov2_ft.train(\n",
+    "    dataset=train_csv,\n",
+    "    validation_set=val_csv,\n",
+    "    output_directory=\"/tmp/results/dinov2_finetuned\",\n",
+    "    skip_save_processed_input=True,\n",
+    ")\n",
+    "\n",
+    "train_time = time.time() - t0\n",
+    "gpu_mem = peak_gpu_mb()\n",
+    "accuracy = evaluate_model(model_dinov2_ft, test_csv)\n",
+    "\n",
+    "results[\"dinov2_finetuned\"] = {\"accuracy\": accuracy, \"train_time_s\": train_time, \"peak_gpu_mb\": gpu_mem}\n",
+    "print(f\"DINOv2 fine-tuned — accuracy: {accuracy:.4f}, time: {train_time:.1f}s, peak GPU: {gpu_mem:.0f}MB\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e7f8a9b0",
+   "metadata": {},
+   "source": [
+    "## Approach 4: CLIP\n",
+    "\n",
+    "CLIP (Radford et al., ICML 2021) from OpenAI is trained on 400M image-text pairs using contrastive learning. Its visual encoder produces embeddings that are aligned with text in a shared latent space.\n",
+    "\n",
+    "CLIP features are particularly useful for:\n",
+    "- Zero-shot image classification (not shown here)\n",
+    "- Tasks where visual-semantic alignment matters\n",
+    "- Multimodal applications\n",
+    "\n",
+    "Default model: `openai/clip-vit-base-patch32` (86M parameters, 768-dim output)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f8a9b0c1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "config_clip = {\n",
+    "    \"model_type\": \"ecd\",\n",
+    "    \"input_features\": [\n",
+    "        {\n",
+    "            \"name\": \"image_path\",\n",
+    "            \"type\": \"image\",\n",
+    "            \"encoder\": {\n",
+    "                \"type\": \"clip\",\n",
+    "                \"use_pretrained\": True,\n",
+    "                \"trainable\": False,\n",
+    "            },\n",
+    "        }\n",
+    "    ],\n",
+    "    \"output_features\": [{\"name\": \"label\", \"type\": \"category\"}],\n",
+    "    \"trainer\": {\"epochs\": 10, \"learning_rate\": 0.001},\n",
+    "}\n",
+    "\n",
+    "reset_gpu()\n",
+    "t0 = time.time()\n",
+    "\n",
+    "model_clip = LudwigModel(config=config_clip, logging_level=30)\n",
+    "_, _, output_dir = model_clip.train(\n",
+    "    dataset=train_csv,\n",
+    "    validation_set=val_csv,\n",
+    "    output_directory=\"/tmp/results/clip\",\n",
+    "    skip_save_processed_input=True,\n",
+    ")\n",
+    "\n",
+    "train_time = time.time() - t0\n",
+    "gpu_mem = peak_gpu_mb()\n",
+    "accuracy = evaluate_model(model_clip, test_csv)\n",
+    "\n",
+    "results[\"clip\"] = {\"accuracy\": accuracy, \"train_time_s\": train_time, \"peak_gpu_mb\": gpu_mem}\n",
+    "print(f\"CLIP linear probe — accuracy: {accuracy:.4f}, time: {train_time:.1f}s, peak GPU: {gpu_mem:.0f}MB\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a9b0c1d2",
+   "metadata": {},
+   "source": [
+    "## Approach 5: SigLIP\n",
+    "\n",
+    "SigLIP (Zhai et al., ICCV 2023) from Google uses sigmoid loss instead of softmax contrastive loss for image-text pre-training. This removes the dependency on global batch statistics and enables better scaling.\n",
+    "\n",
+    "SigLIP features are similar to CLIP in nature but often outperform CLIP on downstream classification tasks, especially with smaller models.\n",
+    "\n",
+    "Default model: `google/siglip-base-patch16-224` (86M parameters, 768-dim output)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b0c1d2e3",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "config_siglip = {\n",
+    "    \"model_type\": \"ecd\",\n",
+    "    \"input_features\": [\n",
+    "        {\n",
+    "            \"name\": \"image_path\",\n",
+    "            \"type\": \"image\",\n",
+    "            \"encoder\": {\n",
+    "                \"type\": \"siglip\",\n",
+    "                \"use_pretrained\": True,\n",
+    "                \"trainable\": False,\n",
+    "            },\n",
+    "        }\n",
+    "    ],\n",
+    "    \"output_features\": [{\"name\": \"label\", \"type\": \"category\"}],\n",
+    "    \"trainer\": {\"epochs\": 10, \"learning_rate\": 0.001},\n",
+    "}\n",
+    "\n",
+    "reset_gpu()\n",
+    "t0 = time.time()\n",
+    "\n",
+    "model_siglip = LudwigModel(config=config_siglip, logging_level=30)\n",
+    "_, _, output_dir = model_siglip.train(\n",
+    "    dataset=train_csv,\n",
+    "    validation_set=val_csv,\n",
+    "    output_directory=\"/tmp/results/siglip\",\n",
+    "    skip_save_processed_input=True,\n",
+    ")\n",
+    "\n",
+    "train_time = time.time() - t0\n",
+    "gpu_mem = peak_gpu_mb()\n",
+    "accuracy = evaluate_model(model_siglip, test_csv)\n",
+    "\n",
+    "results[\"siglip\"] = {\"accuracy\": accuracy, \"train_time_s\": train_time, \"peak_gpu_mb\": gpu_mem}\n",
+    "print(f\"SigLIP linear probe — accuracy: {accuracy:.4f}, time: {train_time:.1f}s, peak GPU: {gpu_mem:.0f}MB\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c1d2e3f4",
+   "metadata": {},
+   "source": [
+    "## Results comparison"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d2e3f4a5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pandas as pd\n",
+    "\n",
+    "rows = []\n",
+    "for name, metrics in results.items():\n",
+    "    rows.append(\n",
+    "        {\n",
+    "            \"Encoder\": name,\n",
+    "            \"Accuracy\": f\"{metrics['accuracy']:.4f}\",\n",
+    "            \"Train time (s)\": f\"{metrics['train_time_s']:.1f}\",\n",
+    "            \"Peak GPU (MB)\": f\"{metrics['peak_gpu_mb']:.0f}\" if metrics[\"peak_gpu_mb\"] > 0 else \"N/A\",\n",
+    "        }\n",
+    "    )\n",
+    "\n",
+    "df_results = pd.DataFrame(rows)\n",
+    "print(df_results.to_string(index=False))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "e3f4a5b6",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Plot accuracy comparison\n",
+    "import matplotlib.pyplot as plt\n",
+    "\n",
+    "names = list(results.keys())\n",
+    "accuracies = [results[n][\"accuracy\"] for n in names]\n",
+    "times = [results[n][\"train_time_s\"] for n in names]\n",
+    "\n",
+    "fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(14, 5))\n",
+    "\n",
+    "colors = [\"#4e79a7\", \"#f28e2b\", \"#e15759\", \"#76b7b2\", \"#59a14f\"]\n",
+    "\n",
+    "bars = ax1.bar(names, accuracies, color=colors[: len(names)])\n",
+    "ax1.set_ylim(0, 1.05)\n",
+    "ax1.set_ylabel(\"Test Accuracy\")\n",
+    "ax1.set_title(\"Accuracy by encoder\")\n",
+    "ax1.tick_params(axis=\"x\", rotation=20)\n",
+    "for bar, acc in zip(bars, accuracies):\n",
+    "    ax1.text(bar.get_x() + bar.get_width() / 2, bar.get_height() + 0.01, f\"{acc:.3f}\", ha=\"center\", va=\"bottom\")\n",
+    "\n",
+    "bars2 = ax2.bar(names, times, color=colors[: len(names)])\n",
+    "ax2.set_ylabel(\"Training time (s)\")\n",
+    "ax2.set_title(\"Training time by encoder\")\n",
+    "ax2.tick_params(axis=\"x\", rotation=20)\n",
+    "for bar, t in zip(bars2, times):\n",
+    "    ax2.text(bar.get_x() + bar.get_width() / 2, bar.get_height() + 1, f\"{t:.0f}s\", ha=\"center\", va=\"bottom\")\n",
+    "\n",
+    "plt.tight_layout()\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f4a5b6c7",
+   "metadata": {},
+   "source": [
+    "## Few-shot: 5 examples per class\n",
+    "\n",
+    "One of the biggest advantages of pretrained encoders is their ability to generalize from very few labeled examples. In this section, we compare `stacked_cnn` vs `dinov2` (linear probe) when trained on only **5 examples per class** (15 examples total).\n",
+    "\n",
+    "This simulates a realistic scenario where annotation is expensive."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a5b6c7d8",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from pathlib import Path\n",
+    "\n",
+    "import pandas as pd\n",
+    "\n",
+    "# Build a 5-shot training set (5 examples per class)\n",
+    "train_df = pd.read_csv(train_csv)\n",
+    "fewshot_df = train_df.groupby(\"label\").sample(n=5, random_state=42).reset_index(drop=True)\n",
+    "\n",
+    "fewshot_csv = \"/tmp/beans/fewshot_5shot.csv\"\n",
+    "fewshot_df.to_csv(fewshot_csv, index=False)\n",
+    "\n",
+    "print(f\"Few-shot training set: {len(fewshot_df)} examples\")\n",
+    "print(fewshot_df[\"label\"].value_counts())"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b6c7d8e9",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "fewshot_results = {}\n",
+    "\n",
+    "# Few-shot with stacked_cnn\n",
+    "reset_gpu()\n",
+    "t0 = time.time()\n",
+    "model_cnn_fs = LudwigModel(config=config_stacked_cnn, logging_level=30)\n",
+    "model_cnn_fs.train(\n",
+    "    dataset=fewshot_csv,\n",
+    "    output_directory=\"/tmp/results/fewshot_cnn\",\n",
+    "    skip_save_processed_input=True,\n",
+    ")\n",
+    "train_time = time.time() - t0\n",
+    "accuracy = evaluate_model(model_cnn_fs, test_csv)\n",
+    "fewshot_results[\"stacked_cnn (5-shot)\"] = {\"accuracy\": accuracy, \"train_time_s\": train_time}\n",
+    "print(f\"stacked_cnn (5-shot) — accuracy: {accuracy:.4f}, time: {train_time:.1f}s\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "c7d8e9f0",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Few-shot with DINOv2 linear probe\n",
+    "reset_gpu()\n",
+    "t0 = time.time()\n",
+    "model_dinov2_fs = LudwigModel(config=config_dinov2_probe, logging_level=30)\n",
+    "model_dinov2_fs.train(\n",
+    "    dataset=fewshot_csv,\n",
+    "    output_directory=\"/tmp/results/fewshot_dinov2\",\n",
+    "    skip_save_processed_input=True,\n",
+    ")\n",
+    "train_time = time.time() - t0\n",
+    "accuracy = evaluate_model(model_dinov2_fs, test_csv)\n",
+    "fewshot_results[\"dinov2_linear_probe (5-shot)\"] = {\"accuracy\": accuracy, \"train_time_s\": train_time}\n",
+    "print(f\"DINOv2 linear probe (5-shot) — accuracy: {accuracy:.4f}, time: {train_time:.1f}s\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d8e9f0a1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "print(\"\\nFew-shot results (5 examples per class):\")\n",
+    "print(\"-\" * 55)\n",
+    "print(f\"{'Encoder':<35} {'Accuracy':>10} {'Time':>8}\")\n",
+    "print(\"-\" * 55)\n",
+    "for name, m in fewshot_results.items():\n",
+    "    print(f\"{name:<35} {m['accuracy']:>10.4f} {m['train_time_s']:>7.1f}s\")\n",
+    "print(\"-\" * 55)\n",
+    "\n",
+    "print(\"\\nFull dataset results (for reference):\")\n",
+    "print(\"-\" * 55)\n",
+    "print(f\"{'Encoder':<35} {'Accuracy':>10} {'Time':>8}\")\n",
+    "print(\"-\" * 55)\n",
+    "for name in [\"stacked_cnn\", \"dinov2_linear_probe\"]:\n",
+    "    m = results[name]\n",
+    "    print(f\"{name:<35} {m['accuracy']:>10.4f} {m['train_time_s']:>7.1f}s\")\n",
+    "print(\"-\" * 55)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e9f0a1b2",
+   "metadata": {},
+   "source": [
+    "## Summary and recommendations\n",
+    "\n",
+    "### When to use each encoder\n",
+    "\n",
+    "| Scenario | Recommended encoder |\n",
+    "|---|---|\n",
+    "| Large dataset (>10k images), custom domain | `stacked_cnn` or fine-tuned pretrained |\n",
+    "| Small dataset (<1k images), natural images | `dinov2` linear probe |\n",
+    "| Few-shot (<50 examples total) | `dinov2` or `siglip` linear probe |\n",
+    "| Multimodal or image-text retrieval | `clip` or `siglip` |\n",
+    "| Best possible accuracy, GPU available | `dinov2` fine-tuned |\n",
+    "\n",
+    "### Key takeaways\n",
+    "\n",
+    "- **Linear probing is surprisingly effective**: freezing the pretrained backbone and training only the head takes a fraction of the time and memory while achieving strong accuracy on small datasets.\n",
+    "- **DINOv2 is the most versatile**: its self-supervised training (no text labels needed) makes it robust across diverse image domains.\n",
+    "- **CLIP and SigLIP are better for semantically rich tasks**: their image-text alignment gives them an edge when classes have meaningful visual-semantic structure.\n",
+    "- **Few-shot gap is dramatic**: with only 5 examples per class, pretrained encoders maintain high accuracy while the CNN from scratch struggles.\n",
+    "\n",
+    "### Changing the pretrained model\n",
+    "\n",
+    "You can use any HuggingFace-compatible model by overriding `pretrained_model_name_or_path`:\n",
+    "\n",
+    "```python\n",
+    "{\n",
+    "    \"type\": \"dinov2\",\n",
+    "    \"use_pretrained\": True,\n",
+    "    \"trainable\": False,\n",
+    "    \"pretrained_model_name_or_path\": \"facebook/dinov2-large\",  # larger model\n",
+    "}\n",
+    "```\n",
+    "\n",
+    "Available DINOv2 variants: `facebook/dinov2-small`, `facebook/dinov2-base`, `facebook/dinov2-large`, `facebook/dinov2-giant`\n",
+    "\n",
+    "Available CLIP variants: `openai/clip-vit-base-patch16`, `openai/clip-vit-large-patch14`\n",
+    "\n",
+    "Available SigLIP variants: `google/siglip-base-patch16-224`, `google/siglip-large-patch16-256`, `google/siglip-so400m-patch14-384`"
+   ]
+  }
+ ],
+ "metadata": {
+  "accelerator": "GPU",
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "name": "python",
+   "version": "3.12.0"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

examples/insurance_lite/config.yaml

@@ -0,0 +1,64 @@
+input_features:
+  - name: image_path
+    type: image
+    preprocessing:
+      height: 224
+      width: 224
+      in_memory: true
+      num_channels: 3
+    encoder: vit
+    img_height: 224
+    in_channels: 3
+    use_pretrained: true
+    hidden_size: 768
+    num_hidden_layers: 12
+    num_attention_heads: 12
+    intermediate_size: 3072
+  - name: insurance_company
+    type: category
+    preprocessing:
+      missing_value_strategy: fill_with_const
+      fill_value: UNKNOWN
+  - name: cost_of_vehicle
+    type: number
+    preprocessing:
+      missing_value_strategy: fill_with_mean
+      normalization: zscore
+  - name: expiry_date
+    type: date
+    preprocessing:
+      missing_value_strategy: fill_with_const
+      fill_value: ""
+      datetime_format: "%Y-%m-%d"
+  - name: min_coverage
+    type: number
+    preprocessing:
+      missing_value_strategy: fill_with_mean
+      normalization: zscore
+  - name: max_coverage
+    type: number
+    preprocessing:
+      missing_value_strategy: fill_with_mean
+      normalization: zscore
+  - name: condition
+    type: category
+    preprocessing:
+      missing_value_strategy: fill_with_const
+      fill_value: UNKNOWN
+combiner:
+  type: concat
+  num_fc_layers: 3
+  output_size: 256
+output_features:
+  - name: amount
+    type: number
+    preprocessing:
+      normalization: zscore
+trainer:
+  epochs: 10
+  early_stop: 0
+  batch_size: 8
+preprocessing:
+  split:
+    type: random
+    probabilities: [0.7, 0.1, 0.2]

examples/insurance_lite/train.py

@@ -0,0 +1,33 @@
+#!/usr/bin/env python
+
+# # Simple Model Training Example on multi-modal data.
+
+# Import required libraries
+import logging
+import os
+import shutil
+
+from ludwig.api import LudwigModel
+from ludwig.datasets import insurance_lite
+
+# clean out prior results
+shutil.rmtree("./results", ignore_errors=True)
+
+# Download and prepare the dataset
+dataset = insurance_lite.load()
+
+# Define Ludwig model object that drive model training
+model = LudwigModel(config="./config.yaml", logging_level=logging.INFO, backend="local")
+
+
+# initiate model training
+(
+    train_stats,  # dictionary containing training statistics
+    preprocessed_data,  # tuple Ludwig Dataset objects of pre-processed training data
+    output_directory,  # location of training results stored on disk
+) = model.train(dataset=dataset, experiment_name="simple_experiment", model_name="simple_model")
+
+# list contents of output directory
+print("contents of output directory:", output_directory)
+for item in os.listdir(output_directory):
+    print("\t", item)

examples/kfold_cv/README.md

@@ -0,0 +1,74 @@
+# K-Ffold Cross Validation Example
+
+This directory contains two examples of performing a k-fold cross validation analysis with Ludwig.
+
+## Classification Example
+
+This example illustrates running the k-fold cv with the `ludwig experiment` cli.
+
+To run this example execute this bash script:
+
+```
+./k-fold_cv_classification.sh
+```
+
+This bash script performs these steps:
+
+- Download and prepare data for training and create a Ludwig config file
+- Execute `ludwig experiment` to run the 5-fold cross validation
+- Display results from the 5-fold cross validation analysis
+
+Sample output:
+
+```
+Cleaning out old results
+Downloading data set
+Preparing data for training
+Saving training and test data sets
+Preparing Ludwig config
+Completed data preparation
+Training: 100%|████████████████████████████████████████████████████████████████████████████████| 12/12 [00:00<00:00, 23.14it/s]
+Evaluation train: 100%|████████████████████████████████████████████████████████████████████████| 12/12 [00:00<00:00, 98.62it/s]
+Evaluation test : 100%|█████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 321.03it/s]
+Training: 100%|███████████████████████████████████████████████████████████████████████████████| 12/12 [00:00<00:00, 190.18it/s]
+Evaluation train: 100%|███████████████████████████████████████████████████████████████████████| 12/12 [00:00<00:00, 331.68it/s]
+Evaluation test : 100%|█████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 298.08it/s]
+<<<< DELETED LINES >>>>>
+Training: 100%|███████████████████████████████████████████████████████████████████████████████| 12/12 [00:00<00:00, 248.00it/s]
+Evaluation train: 100%|███████████████████████████████████████████████████████████████████████| 12/12 [00:00<00:00, 400.31it/s]
+Evaluation test : 100%|█████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 340.35it/s]
+Evaluation: 100%|████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 27.87it/s]
+retrieving results from  results
+#
+# K-fold Cross Validation Results
+#
+{'combined': {'accuracy_mean': 0.9736263736263737,
+              'accuracy_std': 0.011206636293610508,
+              'loss_mean': 0.06359774886251807,
+              'loss_std': 0.011785678840394689},
+ 'diagnosis': {'accuracy_mean': 0.9736263736263737,
+               'accuracy_std': 0.011206636293610508,
+               'average_precision_macro_mean': 0.995842104045726,
+               'average_precision_macro_std': 0.002339014329647542,
+               'average_precision_micro_mean': 0.995842104045726,
+               'average_precision_micro_std': 0.002339014329647542,
+               'average_precision_samples_mean': 0.995842104045726,
+               'average_precision_samples_std': 0.002339014329647542,
+               'loss_mean': 0.06359774886251807,
+               'loss_std': 0.011785678840394689,
+               'roc_auc_macro_mean': 0.9973999160508542,
+               'roc_auc_macro_std': 0.0011259319854886507,
+               'roc_auc_micro_mean': 0.9973999160508542,
+               'roc_auc_micro_std': 0.0011259319854886507}}
+```
+
+## Regression Example
+
+This illustrates using the Ludwig API to run the K-fold cross validation analysis. To run the example, open the jupyter notebook `regression_example.ipynb`. Following steps are performed:
+
+- Download and prepare data for training and create a Ludwig config data structure from a pandas dataframe structure
+- Use `ludwig.api.kfold_cross_validate()` function to run the 5-fold cross validation
+- Display results from the 5-fold cross validation analysis
+
+Expected output from running the example:
+![](../images/regression_kfold_cv_example_results.png)

examples/kfold_cv/display_kfold_cv_results.py

@@ -0,0 +1,33 @@
+#!/usr/bin/env python
+
+
+import argparse
+import os.path
+import pprint
+import sys
+
+from ludwig.utils.data_utils import load_json
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(
+        description="Display K-fold cross validation results",
+        prog="display_kfold_cv_results",
+        usage="%(prog)s [options]",
+    )
+
+    # ----------------------------
+    # Experiment naming parameters
+    # ----------------------------
+    parser.add_argument(
+        "--results_directory", type=str, default="results", help="directory that contains the K-fold cv results"
+    )
+
+    args = parser.parse_args(sys.argv[1:])
+    results_directory = args.results_directory
+
+    print("Retrieving results from ", results_directory)
+
+    kfold_cv_stats = load_json(os.path.join(results_directory, "kfold_training_statistics.json"))
+
+    print("#\n# K-fold Cross Validation Results\n#")
+    pprint.pprint(kfold_cv_stats["overall"])

examples/kfold_cv/k-fold_cv_classification.sh

@@ -0,0 +1,21 @@
+#!/bin/bash
+
+#
+# Download and prepare training data
+#
+python prepare_classification_data_set.py
+
+#
+# Run 5-fold cross validation
+#
+ludwig experiment \
+  --config config.yaml \
+  --dataset data/train.csv \
+  --output_directory results \
+  --logging_level 'error' \
+  -kf 5
+
+#
+# Display results from K-fold cv
+#
+python display_kfold_cv_results.py --results_directory results

examples/kfold_cv/prepare_classification_data_set.py

@@ -0,0 +1,86 @@
+#!/usr/bin/env python
+
+
+# Download and prepare training data set
+# Create Ludwig config file
+#
+# Based on the
+# [UCI Wisconsin Breast Cancer data set](https://archive.ics.uci.edu/ml/datasets/breast+cancer+wisconsin+(original))
+#
+import os.path
+import shutil
+
+import pandas as pd
+import requests
+import yaml
+from sklearn.model_selection import train_test_split
+
+from ludwig.constants import TRAINER
+
+# Constants
+DATA_SET_URL = "https://archive.ics.uci.edu/ml/machine-learning-databases/breast-cancer-wisconsin/wdbc.data"
+DATA_SET = "wdbc.data"
+DATA_DIR = "./data"
+RESULTS_DIR = "results"
+
+# Clean out previous results
+print("Cleaning out old results")
+if os.path.isfile(DATA_SET):
+    os.remove(DATA_SET)
+if os.path.isfile("config.yaml"):
+    os.remove("config.yaml")
+
+shutil.rmtree(RESULTS_DIR, ignore_errors=True)
+shutil.rmtree(DATA_DIR, ignore_errors=True)
+
+# Retrieve data from UCI Machine Learning Repository
+# Download required data
+print("Downloading data set")
+r = requests.get(DATA_SET_URL)
+if r.status_code == 200:
+    with open(DATA_SET, "w") as f:
+        f.write(r.content.decode("utf-8"))
+
+# create pandas dataframe from downloaded data
+print("Preparing data for training")
+raw_df = pd.read_csv(DATA_SET, header=None, sep=",", skipinitialspace=True)
+raw_df.columns = ["ID", "diagnosis"] + ["X" + str(i) for i in range(1, 31)]
+
+# convert diagnosis attribute to binary format
+raw_df["diagnosis"] = raw_df["diagnosis"].map({"M": 1, "B": 0})
+
+# Create train/test split
+print("Saving training and test data sets")
+train_df, test_df = train_test_split(raw_df, train_size=0.8, random_state=17)
+os.mkdir(DATA_DIR)
+train_df.to_csv(os.path.join(DATA_DIR, "train.csv"), index=False)
+test_df.to_csv(os.path.join(DATA_DIR, "test.csv"), index=False)
+
+print("Preparing Ludwig config")
+# Create ludwig input_features
+num_features = ["X" + str(i) for i in range(1, 31)]
+input_features = []
+
+# setup input features for number variables
+for p in num_features:
+    a_feature = {
+        "name": p,
+        "type": "number",
+        "preprocessing": {"missing_value_strategy": "fill_with_mean", "normalization": "zscore"},
+    }
+    input_features.append(a_feature)
+
+# Create ludwig output features
+output_features = [{"name": "diagnosis", "type": "binary", "num_fc_layers": 2, "output_size": 64}]
+
+# setup ludwig config
+config = {
+    "input_features": input_features,
+    "output_features": output_features,
+    TRAINER: {"epochs": 20, "batch_size": 32},
+}
+
+with open("config.yaml", "w") as f:
+    yaml.dump(config, f)
+
+print("Completed data preparation")

examples/lbfgs/config.yaml

@@ -0,0 +1,40 @@
+input_features:
+  - name: RESOURCE
+    type: category
+  - name: MGR_ID
+    type: category
+  - name: ROLE_ROLLUP_1
+    type: category
+  - name: ROLE_ROLLUP_2
+    type: category
+  - name: ROLE_DEPTNAME
+    type: category
+  - name: ROLE_TITLE
+    type: category
+  - name: ROLE_FAMILY_DESC
+    type: category
+  - name: ROLE_FAMILY
+    type: category
+  - name: ROLE_CODE
+    type: category
+output_features:
+  - name: ACTION
+    type: binary
+preprocessing:
+  split:
+    type: fixed
+defaults:
+  category:
+    encoder:
+      type: sparse
+trainer:
+  batch_size: 32769 # entire training set
+  train_steps: 1
+  steps_per_checkpoint: 1
+  learning_rate: 1
+  regularization_lambda: 0.0000057
+  optimizer:
+    type: lbfgs
+    max_iter: 100
+    tolerance_grad: 0.0001
+    history_size: 10

examples/lbfgs/model.py

@@ -0,0 +1,32 @@
+import logging
+
+import pandas as pd
+
+from ludwig.api import LudwigModel
+from ludwig.datasets import amazon_employee_access_challenge
+
+df = amazon_employee_access_challenge.load()
+
+model = LudwigModel(config="config.yaml", logging_level=logging.INFO)
+
+training_statistics, preprocessed_data, output_directory = model.train(
+    df,
+    skip_save_processed_input=True,
+    skip_save_log=True,
+    skip_save_progress=True,
+    skip_save_training_description=True,
+    skip_save_training_statistics=True,
+)
+
+# Predict on unlabeled test
+config = model.config
+config["preprocessing"] = {}
+model.config = config
+unlabeled_test = df[df.split == 2].reset_index(drop=True)
+preds, _ = model.predict(unlabeled_test)
+
+# Save predictions to csv
+action = preds.ACTION_probabilities_True
+submission = pd.merge(unlabeled_test.reset_index(drop=True).id.astype(int), action, left_index=True, right_index=True)
+submission.rename(columns={"ACTION_probabilities_True": "Action", "id": "Id"}, inplace=True)
+submission.to_csv("submission.csv", index=False)

examples/llama2_7b_finetuning_4bit/README.md

@@ -0,0 +1,43 @@
+# Llama2-7b Fine-Tuning 4bit (QLoRA)
+
+[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1c3AO8l_H6V_x37RwQ8V7M6A-RmcBf2tG?usp=sharing)
+
+This example shows how to fine-tune [Llama2-7b](https://huggingface.co/meta-llama/Llama-2-7b-hf) to follow instructions.
+Instruction tuning is the first step in adapting a general purpose Large Language Model into a chatbot.
+
+This example uses no distributed training or big data functionality. It is designed to run locally on any machine
+with GPU availability.
+
+## Prerequisites
+
+- [HuggingFace API Token](https://huggingface.co/docs/hub/security-tokens)
+- Access approval to [Llama2-7b-hf](https://huggingface.co/meta-llama/Llama-2-7b-hf)
+- GPU with at least 12 GiB of VRAM (in our tests, we used an Nvidia T4)
+
+## Running
+
+### Command Line
+
+Set your token environment variable from the terminal, then run the API script:
+
+```bash
+export HUGGING_FACE_HUB_TOKEN="<api_token>"
+./run_train.sh
+```
+
+### Python API
+
+Set your token environment variable from the terminal, then run the API script:
+
+```bash
+export HUGGING_FACE_HUB_TOKEN="<api_token>"
+python train_alpaca.py
+```
+
+## Upload to HuggingFace
+
+You can upload to the HuggingFace Hub from the command line:
+
+```bash
+ludwig upload hf_hub -r <your_org>/<model_name> -m <path/to/model>
+```

examples/llama2_7b_finetuning_4bit/llama2_7b_4bit.yaml

@@ -0,0 +1,28 @@
+model_type: llm
+base_model: meta-llama/Llama-2-7b-hf
+
+quantization:
+  bits: 4
+
+adapter:
+  type: lora
+
+input_features:
+  - name: instruction
+    type: text
+
+output_features:
+  - name: output
+    type: text
+
+trainer:
+  type: finetune
+  learning_rate: 0.0003
+  batch_size: 2
+  gradient_accumulation_steps: 8
+  epochs: 3
+  learning_rate_scheduler:
+    warmup_fraction: 0.01
+
+backend:
+  type: local

examples/llama2_7b_finetuning_4bit/run_train.sh

@@ -0,0 +1,10 @@
+#!/usr/bin/env bash
+
+# Fail fast if an error occurs
+set -e
+
+# Get the directory of this script, which contains the config file
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
+
+# Train
+ludwig train --config ${SCRIPT_DIR}/llama2_7b_4bit.yaml --dataset ludwig://alpaca