chore: import upstream snapshot with attribution
This commit is contained in:
@@ -0,0 +1,23 @@
|
||||
# PostgreSQL
|
||||
POSTGRES_USER=mlflow
|
||||
POSTGRES_PASSWORD=mlflow
|
||||
POSTGRES_DB=mlflow
|
||||
PGPORT=5432
|
||||
|
||||
# S3 Credentials
|
||||
AWS_ACCESS_KEY_ID=s3admin
|
||||
AWS_SECRET_ACCESS_KEY=s3admin
|
||||
AWS_DEFAULT_REGION=us-east-1
|
||||
|
||||
# RustFS
|
||||
RUSTFS_CONSOLE_ENABLE=true
|
||||
S3_BUCKET=mlflow
|
||||
|
||||
# MLflow
|
||||
MLFLOW_VERSION=latest
|
||||
MLFLOW_HOST=0.0.0.0
|
||||
MLFLOW_PORT=5000
|
||||
|
||||
MLFLOW_BACKEND_STORE_URI=postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres:${PGPORT}/${POSTGRES_DB}
|
||||
MLFLOW_ARTIFACTS_DESTINATION=s3://${S3_BUCKET}
|
||||
MLFLOW_S3_ENDPOINT_URL=http://storage:9000
|
||||
@@ -0,0 +1,239 @@
|
||||
# MLflow with Docker Compose (PostgreSQL + S3-Compatible Storage)
|
||||
|
||||
This directory provides a **Docker Compose** setup for running **MLflow** locally with a **PostgreSQL** backend store and **RustFS** for S3-compatible artifact storage.
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
- **MLflow Tracking Server**
|
||||
Serves the REST API and UI (default: `http://localhost:5000`).
|
||||
|
||||
- **PostgreSQL**
|
||||
Stores MLflow's metadata (experiments, runs, params, metrics).
|
||||
|
||||
- **RustFS Artifact Storage**
|
||||
Stores model files and run artifacts.
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **Git**
|
||||
- **Docker** and **Docker Compose**
|
||||
- On macOS/Windows: Docker Desktop
|
||||
- On Linux: Docker Engine + compose plugin
|
||||
|
||||
Verify installation:
|
||||
|
||||
```bash
|
||||
docker --version
|
||||
docker compose version
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 1. Clone the Repository
|
||||
|
||||
```bash
|
||||
git clone https://github.com/mlflow/mlflow.git
|
||||
cd mlflow/docker-compose/
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Configure Environment
|
||||
|
||||
Copy and customize the environment file:
|
||||
|
||||
```bash
|
||||
cp .env.dev.example .env
|
||||
```
|
||||
|
||||
The `.env` file defines:
|
||||
|
||||
- MLflow server port
|
||||
- PostgreSQL credentials
|
||||
- S3 bucket name
|
||||
- S3-compatible endpoint URL
|
||||
- Backend-specific configuration for RustFS
|
||||
|
||||
**Common variables**:
|
||||
|
||||
- **PostgreSQL**
|
||||
|
||||
- `POSTGRES_USER=mlflow`
|
||||
- `POSTGRES_PASSWORD=mlflow`
|
||||
- `POSTGRES_DB=mlflow`
|
||||
|
||||
- **S3**
|
||||
|
||||
- `AWS_ACCESS_KEY_ID=s3admin`
|
||||
- `AWS_SECRET_ACCESS_KEY=s3admin`
|
||||
- `AWS_DEFAULT_REGION=us-east-1`
|
||||
- `S3_BUCKET=mlflow`
|
||||
|
||||
- **RustFS**
|
||||
|
||||
- `RUSTFS_CONSOLE_ENABLE=true`
|
||||
|
||||
- **MLflow**
|
||||
- `MLFLOW_VERSION=latest`
|
||||
- `MLFLOW_HOST=0.0.0.0`
|
||||
- `MLFLOW_PORT=5000`
|
||||
- `MLFLOW_BACKEND_STORE_URI=postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB}`
|
||||
- `MLFLOW_ARTIFACTS_DESTINATION=s3://${S3_BUCKET}`
|
||||
- `MLFLOW_S3_ENDPOINT_URL=http://storage:9000`
|
||||
|
||||
---
|
||||
|
||||
## 3. Launch the Stack
|
||||
|
||||
Inside directory **mlflow/docker-compose**:
|
||||
|
||||
```bash
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
This will:
|
||||
|
||||
- Start PostgreSQL
|
||||
- Start RustFS
|
||||
- Start MLflow
|
||||
- Create the S3 bucket if it doesn't exist
|
||||
|
||||
Check status:
|
||||
|
||||
```bash
|
||||
docker compose ps
|
||||
```
|
||||
|
||||
Tail logs:
|
||||
|
||||
```bash
|
||||
docker compose logs -f
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Access MLflow
|
||||
|
||||
Once running:
|
||||
|
||||
- Open `http://localhost:5000` (or the port defined in `.env`)
|
||||
|
||||
You can now log runs, metrics, artifacts, and models to your local MLflow instance.
|
||||
|
||||
---
|
||||
|
||||
## 5. Shutdown
|
||||
|
||||
To stop and remove containers:
|
||||
|
||||
```bash
|
||||
docker compose down
|
||||
```
|
||||
|
||||
To reset everything, including volumes:
|
||||
|
||||
```bash
|
||||
docker compose down -v
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Tips & Troubleshooting
|
||||
|
||||
### RustFS Notes (important)
|
||||
|
||||
- Set **server domains/host** so virtual-hosted requests can be resolved by RustFS:
|
||||
|
||||
```env
|
||||
RUSTFS_SERVER_DOMAINS=storage:9000
|
||||
```
|
||||
|
||||
(match the compose service DNS name)
|
||||
|
||||
- Prefer AWS CLI **`s3api`** for bucket creation. Some S3 clients default to **path-style** on custom endpoints; if bucket creation fails with `InvalidBucketName`, switch to `s3api` or a client like MinIO `mc`.
|
||||
|
||||
- Inside MLflow, use the internal endpoint:
|
||||
```env
|
||||
MLFLOW_S3_ENDPOINT_URL=http://storage:9000
|
||||
MLFLOW_ARTIFACTS_DESTINATION=s3://mlflow/
|
||||
```
|
||||
|
||||
### Healthcheck Example
|
||||
|
||||
RustFS usually responds on `/health` with a json that contains the status of the server:
|
||||
|
||||
```sh
|
||||
curl -s http://127.0.0.1:9000/health | grep -q '\"status\"\\s*:\\s*\"ok\"'
|
||||
```
|
||||
|
||||
Use that in a container healthcheck (no `-f`, 4xx may appear during bootstrap).
|
||||
|
||||
---
|
||||
|
||||
### Artifact Upload Issues
|
||||
|
||||
Verify:
|
||||
|
||||
- `MLFLOW_ARTIFACTS_DESTINATION=s3://<bucket>/`
|
||||
- `MLFLOW_S3_ENDPOINT_URL=http://<service>:<port>`
|
||||
- AWS credentials match the backend configuration
|
||||
|
||||
To verify S3 storage is working:
|
||||
|
||||
```bash
|
||||
aws --endpoint-url=${MLFLOW_S3_ENDPOINT_URL} s3api list-buckets
|
||||
|
||||
echo hi > /tmp/t.txt
|
||||
aws --endpoint-url=${MLFLOW_S3_ENDPOINT_URL} s3 cp /tmp/t.txt s3://${S3_BUCKET}/t.txt
|
||||
aws --endpoint-url=${MLFLOW_S3_ENDPOINT_URL} s3 cp s3://${S3_BUCKET}/t.txt -
|
||||
```
|
||||
|
||||
If this passes, MLflow can read and write artifacts to RustFS.
|
||||
|
||||
### Troubleshooting
|
||||
|
||||
- `InvalidBucketName` on create-bucket → use `s3api` (virtual-host friendly) or MinIO `mc`; ensure `RUSTFS_SERVER_DOMAINS` matches the S3 hostname.
|
||||
- Endpoint issues from MLflow → make sure `MLFLOW_S3_ENDPOINT_URL` uses the **service name** visible from MLflow (e.g., `http://storage:9000`).
|
||||
|
||||
### Resetting the Environment
|
||||
|
||||
```bash
|
||||
docker compose down -v
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
### Logs
|
||||
|
||||
```bash
|
||||
docker compose logs -f mlflow
|
||||
docker compose logs -f postgres
|
||||
docker compose logs -f storage
|
||||
```
|
||||
|
||||
### Port Conflicts
|
||||
|
||||
Edit `.env` and restart containers:
|
||||
|
||||
```bash
|
||||
docker compose down
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
- Point your training scripts to this server:
|
||||
```bash
|
||||
export MLFLOW_TRACKING_URI=http://localhost:5000
|
||||
```
|
||||
- Start logging runs with `mlflow.start_run()` (Python) or the MLflow CLI.
|
||||
- Customize the `.env` and `docker-compose.yml` to fit your local workflow (e.g., change image tags, add volumes, etc.).
|
||||
|
||||
---
|
||||
|
||||
**You now have a fully local MLflow stack with persistent metadata and artifact storage—ideal for development and experimentation.**
|
||||
@@ -0,0 +1,125 @@
|
||||
volumes:
|
||||
pgdata:
|
||||
storage-data:
|
||||
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:15
|
||||
container_name: mlflow-postgres
|
||||
environment:
|
||||
POSTGRES_USER: ${POSTGRES_USER}
|
||||
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
|
||||
POSTGRES_DB: ${POSTGRES_DB}
|
||||
PGPORT: ${PGPORT}
|
||||
volumes:
|
||||
- pgdata:/var/lib/postgresql/data
|
||||
ports:
|
||||
- ${PGPORT}:${PGPORT}
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER} -d ${POSTGRES_DB} -p ${PGPORT}"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 10
|
||||
|
||||
storage:
|
||||
image: rustfs/rustfs:1.0.0-alpha.83
|
||||
container_name: storage
|
||||
environment:
|
||||
RUSTFS_ADDRESS: :9000
|
||||
RUSTFS_SERVER_DOMAINS: storage:9000
|
||||
RUSTFS_REGION: ${AWS_DEFAULT_REGION:-us-east-1}
|
||||
RUSTFS_ACCESS_KEY: ${AWS_ACCESS_KEY_ID:-s3admin}
|
||||
RUSTFS_SECRET_KEY: ${AWS_SECRET_ACCESS_KEY:-s3admin}
|
||||
RUSTFS_CONSOLE_ENABLE: ${RUSTFS_CONSOLE_ENABLE:-true}
|
||||
ports:
|
||||
- "9000:9000"
|
||||
- "9001:9001"
|
||||
volumes:
|
||||
- storage-data:/data
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", 'curl -s http://127.0.0.1:9000/health | grep -q ''"status":"ok"''']
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
start_period: 10s
|
||||
|
||||
create-bucket:
|
||||
image: amazon/aws-cli:2.33.25
|
||||
container_name: mlflow-create-bucket
|
||||
depends_on:
|
||||
storage:
|
||||
condition: service_healthy
|
||||
entrypoint: >
|
||||
/bin/sh -c "
|
||||
set -e;
|
||||
echo 'Waiting for S3 gateway getting ready...';
|
||||
if aws --endpoint-url=${MLFLOW_S3_ENDPOINT_URL} s3api head-bucket --bucket ${S3_BUCKET} 2>/dev/null; then
|
||||
echo 'Bucket ${S3_BUCKET} already exists. Skipping creation.';
|
||||
else
|
||||
echo 'Creating bucket ${S3_BUCKET}...';
|
||||
aws --endpoint-url=${MLFLOW_S3_ENDPOINT_URL} s3api create-bucket --bucket ${S3_BUCKET} --region ${AWS_DEFAULT_REGION};
|
||||
fi
|
||||
"
|
||||
environment:
|
||||
AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
|
||||
AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
|
||||
AWS_DEFAULT_REGION: ${AWS_DEFAULT_REGION}
|
||||
AWS_S3_ADDRESSING_STYLE: path
|
||||
MLFLOW_S3_ENDPOINT_URL: ${MLFLOW_S3_ENDPOINT_URL}
|
||||
S3_BUCKET: ${S3_BUCKET}
|
||||
restart: "no"
|
||||
|
||||
mlflow:
|
||||
image: ghcr.io/mlflow/mlflow:${MLFLOW_VERSION}
|
||||
container_name: mlflow-server
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
storage:
|
||||
condition: service_healthy
|
||||
create-bucket:
|
||||
condition: service_completed_successfully
|
||||
environment:
|
||||
# Backend store URI built from vars
|
||||
MLFLOW_BACKEND_STORE_URI: ${MLFLOW_BACKEND_STORE_URI}
|
||||
|
||||
# S3/RustFS settings
|
||||
MLFLOW_S3_ENDPOINT_URL: ${MLFLOW_S3_ENDPOINT_URL}
|
||||
MLFLOW_ARTIFACTS_DESTINATION: ${MLFLOW_ARTIFACTS_DESTINATION}
|
||||
AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
|
||||
AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
|
||||
AWS_DEFAULT_REGION: ${AWS_DEFAULT_REGION}
|
||||
MLFLOW_S3_IGNORE_TLS: "true"
|
||||
|
||||
# Server host/port
|
||||
MLFLOW_HOST: ${MLFLOW_HOST}
|
||||
MLFLOW_PORT: ${MLFLOW_PORT}
|
||||
command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- |
|
||||
pip install --no-cache-dir psycopg2-binary boto3
|
||||
mlflow server \
|
||||
--backend-store-uri "${MLFLOW_BACKEND_STORE_URI}" \
|
||||
--artifacts-destination "${MLFLOW_ARTIFACTS_DESTINATION}" \
|
||||
--serve-artifacts \
|
||||
--host "${MLFLOW_HOST}" \
|
||||
--port "${MLFLOW_PORT}"
|
||||
ports:
|
||||
- "${MLFLOW_PORT}:${MLFLOW_PORT}"
|
||||
healthcheck:
|
||||
test:
|
||||
[
|
||||
"CMD",
|
||||
"python",
|
||||
"-c",
|
||||
"import urllib.request; urllib.request.urlopen('http://localhost:${MLFLOW_PORT}/health')",
|
||||
]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 30
|
||||
|
||||
networks:
|
||||
default:
|
||||
name: mlflow-network
|
||||
Reference in New Issue
Block a user