chore: import upstream snapshot with attribution

This commit is contained in:
wehub-resource-sync
2026-07-13 13:22:34 +08:00
commit 4b22cfda96
9037 changed files with 2363717 additions and 0 deletions
+23
View File
@@ -0,0 +1,23 @@
# PostgreSQL
POSTGRES_USER=mlflow
POSTGRES_PASSWORD=mlflow
POSTGRES_DB=mlflow
PGPORT=5432
# S3 Credentials
AWS_ACCESS_KEY_ID=s3admin
AWS_SECRET_ACCESS_KEY=s3admin
AWS_DEFAULT_REGION=us-east-1
# RustFS
RUSTFS_CONSOLE_ENABLE=true
S3_BUCKET=mlflow
# MLflow
MLFLOW_VERSION=latest
MLFLOW_HOST=0.0.0.0
MLFLOW_PORT=5000
MLFLOW_BACKEND_STORE_URI=postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres:${PGPORT}/${POSTGRES_DB}
MLFLOW_ARTIFACTS_DESTINATION=s3://${S3_BUCKET}
MLFLOW_S3_ENDPOINT_URL=http://storage:9000
+239
View File
@@ -0,0 +1,239 @@
# MLflow with Docker Compose (PostgreSQL + S3-Compatible Storage)
This directory provides a **Docker Compose** setup for running **MLflow** locally with a **PostgreSQL** backend store and **RustFS** for S3-compatible artifact storage.
---
## Overview
- **MLflow Tracking Server**
Serves the REST API and UI (default: `http://localhost:5000`).
- **PostgreSQL**
Stores MLflow's metadata (experiments, runs, params, metrics).
- **RustFS Artifact Storage**
Stores model files and run artifacts.
---
## Prerequisites
- **Git**
- **Docker** and **Docker Compose**
- On macOS/Windows: Docker Desktop
- On Linux: Docker Engine + compose plugin
Verify installation:
```bash
docker --version
docker compose version
```
---
## 1. Clone the Repository
```bash
git clone https://github.com/mlflow/mlflow.git
cd mlflow/docker-compose/
```
---
## 2. Configure Environment
Copy and customize the environment file:
```bash
cp .env.dev.example .env
```
The `.env` file defines:
- MLflow server port
- PostgreSQL credentials
- S3 bucket name
- S3-compatible endpoint URL
- Backend-specific configuration for RustFS
**Common variables**:
- **PostgreSQL**
- `POSTGRES_USER=mlflow`
- `POSTGRES_PASSWORD=mlflow`
- `POSTGRES_DB=mlflow`
- **S3**
- `AWS_ACCESS_KEY_ID=s3admin`
- `AWS_SECRET_ACCESS_KEY=s3admin`
- `AWS_DEFAULT_REGION=us-east-1`
- `S3_BUCKET=mlflow`
- **RustFS**
- `RUSTFS_CONSOLE_ENABLE=true`
- **MLflow**
- `MLFLOW_VERSION=latest`
- `MLFLOW_HOST=0.0.0.0`
- `MLFLOW_PORT=5000`
- `MLFLOW_BACKEND_STORE_URI=postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB}`
- `MLFLOW_ARTIFACTS_DESTINATION=s3://${S3_BUCKET}`
- `MLFLOW_S3_ENDPOINT_URL=http://storage:9000`
---
## 3. Launch the Stack
Inside directory **mlflow/docker-compose**:
```bash
docker compose up -d
```
This will:
- Start PostgreSQL
- Start RustFS
- Start MLflow
- Create the S3 bucket if it doesn't exist
Check status:
```bash
docker compose ps
```
Tail logs:
```bash
docker compose logs -f
```
---
## 4. Access MLflow
Once running:
- Open `http://localhost:5000` (or the port defined in `.env`)
You can now log runs, metrics, artifacts, and models to your local MLflow instance.
---
## 5. Shutdown
To stop and remove containers:
```bash
docker compose down
```
To reset everything, including volumes:
```bash
docker compose down -v
```
---
## Tips & Troubleshooting
### RustFS Notes (important)
- Set **server domains/host** so virtual-hosted requests can be resolved by RustFS:
```env
RUSTFS_SERVER_DOMAINS=storage:9000
```
(match the compose service DNS name)
- Prefer AWS CLI **`s3api`** for bucket creation. Some S3 clients default to **path-style** on custom endpoints; if bucket creation fails with `InvalidBucketName`, switch to `s3api` or a client like MinIO `mc`.
- Inside MLflow, use the internal endpoint:
```env
MLFLOW_S3_ENDPOINT_URL=http://storage:9000
MLFLOW_ARTIFACTS_DESTINATION=s3://mlflow/
```
### Healthcheck Example
RustFS usually responds on `/health` with a json that contains the status of the server:
```sh
curl -s http://127.0.0.1:9000/health | grep -q '\"status\"\\s*:\\s*\"ok\"'
```
Use that in a container healthcheck (no `-f`, 4xx may appear during bootstrap).
---
### Artifact Upload Issues
Verify:
- `MLFLOW_ARTIFACTS_DESTINATION=s3://<bucket>/`
- `MLFLOW_S3_ENDPOINT_URL=http://<service>:<port>`
- AWS credentials match the backend configuration
To verify S3 storage is working:
```bash
aws --endpoint-url=${MLFLOW_S3_ENDPOINT_URL} s3api list-buckets
echo hi > /tmp/t.txt
aws --endpoint-url=${MLFLOW_S3_ENDPOINT_URL} s3 cp /tmp/t.txt s3://${S3_BUCKET}/t.txt
aws --endpoint-url=${MLFLOW_S3_ENDPOINT_URL} s3 cp s3://${S3_BUCKET}/t.txt -
```
If this passes, MLflow can read and write artifacts to RustFS.
### Troubleshooting
- `InvalidBucketName` on create-bucket → use `s3api` (virtual-host friendly) or MinIO `mc`; ensure `RUSTFS_SERVER_DOMAINS` matches the S3 hostname.
- Endpoint issues from MLflow → make sure `MLFLOW_S3_ENDPOINT_URL` uses the **service name** visible from MLflow (e.g., `http://storage:9000`).
### Resetting the Environment
```bash
docker compose down -v
docker compose up -d
```
### Logs
```bash
docker compose logs -f mlflow
docker compose logs -f postgres
docker compose logs -f storage
```
### Port Conflicts
Edit `.env` and restart containers:
```bash
docker compose down
docker compose up -d
```
---
## Next Steps
- Point your training scripts to this server:
```bash
export MLFLOW_TRACKING_URI=http://localhost:5000
```
- Start logging runs with `mlflow.start_run()` (Python) or the MLflow CLI.
- Customize the `.env` and `docker-compose.yml` to fit your local workflow (e.g., change image tags, add volumes, etc.).
---
**You now have a fully local MLflow stack with persistent metadata and artifact storage—ideal for development and experimentation.**
+125
View File
@@ -0,0 +1,125 @@
volumes:
pgdata:
storage-data:
services:
postgres:
image: postgres:15
container_name: mlflow-postgres
environment:
POSTGRES_USER: ${POSTGRES_USER}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
POSTGRES_DB: ${POSTGRES_DB}
PGPORT: ${PGPORT}
volumes:
- pgdata:/var/lib/postgresql/data
ports:
- ${PGPORT}:${PGPORT}
healthcheck:
test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER} -d ${POSTGRES_DB} -p ${PGPORT}"]
interval: 5s
timeout: 3s
retries: 10
storage:
image: rustfs/rustfs:1.0.0-alpha.83
container_name: storage
environment:
RUSTFS_ADDRESS: :9000
RUSTFS_SERVER_DOMAINS: storage:9000
RUSTFS_REGION: ${AWS_DEFAULT_REGION:-us-east-1}
RUSTFS_ACCESS_KEY: ${AWS_ACCESS_KEY_ID:-s3admin}
RUSTFS_SECRET_KEY: ${AWS_SECRET_ACCESS_KEY:-s3admin}
RUSTFS_CONSOLE_ENABLE: ${RUSTFS_CONSOLE_ENABLE:-true}
ports:
- "9000:9000"
- "9001:9001"
volumes:
- storage-data:/data
restart: unless-stopped
healthcheck:
test: ["CMD-SHELL", 'curl -s http://127.0.0.1:9000/health | grep -q ''"status":"ok"''']
interval: 10s
timeout: 5s
retries: 5
start_period: 10s
create-bucket:
image: amazon/aws-cli:2.33.25
container_name: mlflow-create-bucket
depends_on:
storage:
condition: service_healthy
entrypoint: >
/bin/sh -c "
set -e;
echo 'Waiting for S3 gateway getting ready...';
if aws --endpoint-url=${MLFLOW_S3_ENDPOINT_URL} s3api head-bucket --bucket ${S3_BUCKET} 2>/dev/null; then
echo 'Bucket ${S3_BUCKET} already exists. Skipping creation.';
else
echo 'Creating bucket ${S3_BUCKET}...';
aws --endpoint-url=${MLFLOW_S3_ENDPOINT_URL} s3api create-bucket --bucket ${S3_BUCKET} --region ${AWS_DEFAULT_REGION};
fi
"
environment:
AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
AWS_DEFAULT_REGION: ${AWS_DEFAULT_REGION}
AWS_S3_ADDRESSING_STYLE: path
MLFLOW_S3_ENDPOINT_URL: ${MLFLOW_S3_ENDPOINT_URL}
S3_BUCKET: ${S3_BUCKET}
restart: "no"
mlflow:
image: ghcr.io/mlflow/mlflow:${MLFLOW_VERSION}
container_name: mlflow-server
depends_on:
postgres:
condition: service_healthy
storage:
condition: service_healthy
create-bucket:
condition: service_completed_successfully
environment:
# Backend store URI built from vars
MLFLOW_BACKEND_STORE_URI: ${MLFLOW_BACKEND_STORE_URI}
# S3/RustFS settings
MLFLOW_S3_ENDPOINT_URL: ${MLFLOW_S3_ENDPOINT_URL}
MLFLOW_ARTIFACTS_DESTINATION: ${MLFLOW_ARTIFACTS_DESTINATION}
AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
AWS_DEFAULT_REGION: ${AWS_DEFAULT_REGION}
MLFLOW_S3_IGNORE_TLS: "true"
# Server host/port
MLFLOW_HOST: ${MLFLOW_HOST}
MLFLOW_PORT: ${MLFLOW_PORT}
command:
- /bin/bash
- -c
- |
pip install --no-cache-dir psycopg2-binary boto3
mlflow server \
--backend-store-uri "${MLFLOW_BACKEND_STORE_URI}" \
--artifacts-destination "${MLFLOW_ARTIFACTS_DESTINATION}" \
--serve-artifacts \
--host "${MLFLOW_HOST}" \
--port "${MLFLOW_PORT}"
ports:
- "${MLFLOW_PORT}:${MLFLOW_PORT}"
healthcheck:
test:
[
"CMD",
"python",
"-c",
"import urllib.request; urllib.request.urlopen('http://localhost:${MLFLOW_PORT}/health')",
]
interval: 10s
timeout: 5s
retries: 30
networks:
default:
name: mlflow-network