tracer-cloud--opensre/tests/e2e/ml_recommendations_service/README.md

Here’s how to build a **simple MVP** of this Dressy pipeline on AWS with the smallest surface area, while keeping the same core shape: ingest product data, build customer profiles, train a model, serve it, and return recommendations.

## Business Objective

Ship an end to end recommender MVP that incorporates new products into recommendations within 12 hours.

## Scope

Product ingest and labeling, purchase event capture, nightly feature build, nightly training, versioned model deploy, online inference, basic monitoring.

## MVP Architecture on AWS

### 1) Ingest product uploads

**Goal:** get new dresses into a canonical store quickly.

* **S3** as the landing zone: `s3://dressy-raw/products/`
* Vendor uploads JSON + images to S3.
* **S3 event notification → Lambda** writes a row into a “products” table and enqueues a labeling job.

MVP stores:

* **DynamoDB** `Products` table for serving-time reads (fast, cheap).
* Optional: also dump raw events to **S3** for replay.

### 2) Product labeling pipeline (lightweight)

**Goal:** normalize messy vendor data into consistent attributes.

MVP approach:

* Lambda takes the image, calls **Amazon Rekognition** (labels, colors, maybe text), merges vendor metadata.
* Writes labeled product record to DynamoDB `Products` and to S3 `s3://dressy-features/products/` as parquet or JSONL.

Why Rekognition: zero infra, good enough, fast.

### 3) Capture purchase events

**Goal:** build user history.

* Frontend calls **API Gateway → Lambda** `POST /purchase`
* Lambda writes:

  * raw event to **Kinesis Data Firehose → S3** `s3://dressy-raw/purchases/`
  * and an aggregated record to DynamoDB `UserProfiles` (append or counters)

MVP: DynamoDB is your online feature store. S3 is your offline truth.

### 4) Build customer profiles and training dataset (nightly)

**Goal:** turn raw events into training examples.

* **EventBridge scheduled rule** runs nightly.
* Job runs in **AWS Glue** (Spark) or **ECS Fargate** container.

  * Reads `products` and `purchases` from S3
  * Produces:

    * `s3://dressy-features/user_profiles/`
    * `s3://dressy-training/training_set/`
  * Optionally updates DynamoDB `UserProfiles` in batch.

Pick one:

* Glue if you want managed ETL.
* ECS Fargate if you want simple Python you control.

### 5) Train model (nightly)

**Goal:** produce a new model artifact.

* Use **SageMaker Training Job** triggered after the dataset build.
* Training code reads from `s3://dressy-training/…`
* Outputs model to `s3://dressy-model-registry/models/<date or version>/`

For the recommender itself, keep it stupid-simple for MVP:

* Start with **two-tower retrieval** or even **matrix factorization**.
* Or go even simpler: build embeddings and do nearest neighbor search.

### 6) Evaluate and promote

**Goal:** prevent obviously bad models from shipping.

* After training, run a small **SageMaker Processing Job** or Lambda that:

  * computes a few metrics on a holdout set (top-k hit rate, precision@k)
  * writes metrics JSON to S3
  * decides whether to “promote”

Promotion mechanism MVP:

* Update an S3 pointer: `s3://dressy-model-registry/production/latest.tar.gz`
* Or store “production version” in **SSM Parameter Store**.

### 7) Serve recommendations (online)

**Goal:** low latency inference.

Two MVP options:

**Option A: SageMaker real-time endpoint**

* Deploy “latest” model to a **SageMaker Endpoint**.
* **Recommendations API** (API Gateway → Lambda) calls the endpoint.
* Lambda enriches with product details from DynamoDB `Products`.

**Option B: No SageMaker endpoint, just Lambda**

* If model is tiny (embeddings + ANN index), you can load it in Lambda from S3 at cold start.
* Use **OpenSearch k-NN** as your vector index (store product embeddings).
* Lambda query: user embedding → OpenSearch → top-k product IDs → DynamoDB lookup.

If you want minimum moving parts, start with **Option A**.

### 8) Frontend purchase service

* The “Purchasing Service” is basically your main backend.
* It calls:

  * `GET /recommendations?user_id=...`
  * `POST /purchase`

## Implementation Plan

### Phase 1: Working loop in 1 day

* S3 product upload
* Lambda labeling (Rekognition) → DynamoDB Products
* Purchase API → Firehose → S3 + DynamoDB UserProfiles
* Recommendations API returns “most popular” or “recently added”
  This gives you the plumbing and the UI integration.

### Phase 2: Nightly training in 2 to 3 days

* Nightly ETL to build training set
* SageMaker training job outputs model
* Basic evaluation + promotion
* SageMaker endpoint serving

### Phase 3: Make it not fragile in 1 to 2 days

* Add retries, DLQs, idempotency keys for purchase events
* CloudWatch dashboards and alarms
* Canary deploy for new model versions

## Success Criteria

* New product upload appears in recommendation candidates within 12 hours.
* Recommendations endpoint p95 latency under 300ms.
* Model deploy is versioned and reversible in under 5 minutes.
* At least one end to end alarm catches a broken pipeline before users notice.

## Out of Scope

* Near-real-time retraining
* Complex feature store governance
* Multi-region serving
* Human labeling workflows

If you tell me one thing: do you want “MVP” to mean **fewest AWS services** or **most managed**? I’ll lock the plan to a single stack (either “all serverless” or “SageMaker-centric”) and give you exact resources, names, and the event flow.