wehub-resource-sync
72 lines
3.7 KiB
ReStructuredText
72 lines
3.7 KiB
ReStructuredText
Dockerized Model Training with MLflow
|
|
-------------------------------------
|
|
This directory contains an MLflow project that trains a linear regression model on the UC Irvine
|
|
Wine Quality Dataset. The project uses a Docker image to capture the dependencies needed to run
|
|
training code. Running a project in a Docker environment (as opposed to Conda) allows for capturing
|
|
non-Python dependencies, e.g. Java libraries. In the future, we also hope to add tools to MLflow
|
|
for running Dockerized projects e.g. on a Kubernetes cluster for scale out.
|
|
|
|
Structure of this MLflow Project
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
This MLflow project contains a ``train.py`` file that trains a scikit-learn model and uses
|
|
MLflow Tracking APIs to log the model and its metadata (e.g., hyperparameters and metrics)
|
|
for later use and reference. ``train.py`` operates on the Wine Quality Dataset, which is included
|
|
in ``wine-quality.csv``.
|
|
|
|
Most importantly, the project also includes an ``MLproject`` file, which specifies the Docker
|
|
container environment in which to run the project using the ``docker_env`` field:
|
|
|
|
.. code-block:: yaml
|
|
|
|
docker_env:
|
|
image: mlflow-docker-example
|
|
|
|
Here, ``image`` can be any valid argument to ``docker run``, such as the tag, ID or URL of a Docker
|
|
image (see `Docker docs <https://docs.docker.com/engine/reference/run/#general-form>`_). The above
|
|
example references a locally-stored image (``mlflow-docker-example``) by tag.
|
|
|
|
Finally, the project includes a ``Dockerfile`` that is used to build the image referenced by the
|
|
``MLproject`` file. The ``Dockerfile`` specifies library dependencies required by the project, such
|
|
as ``mlflow`` and ``scikit-learn``.
|
|
|
|
Running this Example
|
|
^^^^^^^^^^^^^^^^^^^^
|
|
|
|
First, install MLflow (via ``pip install mlflow``) and install
|
|
`Docker <https://www.docker.com/get-started>`_.
|
|
|
|
Then, build the image for the project's Docker container environment. You must use the same image
|
|
name that is given by the ``docker_env.image`` field of the MLproject file. In this example, the
|
|
image name is ``mlflow-docker-example``. Issue the following command to build an image with this
|
|
name:
|
|
|
|
.. code-block:: bash
|
|
|
|
docker build -t mlflow-docker-example -f Dockerfile .
|
|
|
|
Note that the name if the image used in the ``docker build`` command, ``mlflow-docker-example``,
|
|
matches the name of the image referenced in the ``MLproject`` file.
|
|
|
|
Finally, run the example project using ``mlflow run examples/docker -P alpha=0.5``.
|
|
|
|
.. note::
|
|
If running this example on a Mac with Apple silicon, ensure that Docker Desktop is running and
|
|
that you are logged in to the Docker Desktop service.
|
|
If you are modifying the example ``DockerFile`` to specify older versions of ``scikit-learn``,
|
|
you should enable `Rosetta compatibility <https://docs.docker.com/desktop/settings/mac/#features-in-development>`_
|
|
in the Docker Desktop configuration settings to ensure that the appropriate ``cython`` compiler is used.
|
|
|
|
What happens when the project is run?
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
Running ``mlflow run examples/docker`` builds a new Docker image based on ``mlflow-docker-example``
|
|
that also contains our project code. The resulting image is tagged as
|
|
``mlflow-docker-example-<git-version>`` where ``<git-version>`` is the git commit ID. After the image is
|
|
built, MLflow executes the default (main) project entry point within the container using ``docker run``.
|
|
|
|
Environment variables, such as ``MLFLOW_TRACKING_URI``, are propagated inside the container during
|
|
project execution. When running against a local tracking URI, MLflow mounts the host system's
|
|
tracking directory (e.g., a local ``mlruns`` directory) inside the container so that metrics and
|
|
params logged during project execution are accessible afterwards.
|