wehub/ray-project--ray
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 13:17:40 +08:00
commit f1825c8ceb
10096 changed files with 2364182 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/source/serve/tutorials/video-analysis/README.ipynb
+1411
View File
File diff suppressed because it is too large Load Diff
doc/source/serve/tutorials/video-analysis/README.md
+724
View File
@@ -0,0 +1,724 @@
---
orphan: true
---
# Video analysis inference pipeline with Ray Serve
This notebook demonstrates how to build a production-grade video analysis pipeline with [Ray Serve](https://docs.ray.io/en/latest/serve/). The pipeline processes videos from S3 and extracts:
- **Tags**: [Zero-shot classification](https://huggingface.co/tasks/zero-shot-image-classification) labels (such as "kitchen", "office", "park")
- **Captions**: Retrieval-based descriptions matching the video content
- **Scene changes**: Detected transitions using [exponential moving average (EMA)](https://en.wikipedia.org/wiki/Exponential_smoothing) analysis
The system uses [**SigLIP**](https://huggingface.co/google/siglip-so400m-patch14-384) (`google/siglip-so400m-patch14-384`) as the vision-language backbone. SigLIP provides a unified embedding space for both images and text, enabling zero-shot classification and retrieval without task-specific fine-tuning.
## Architecture
The pipeline splits work across three [Ray Serve deployments](https://docs.ray.io/en/latest/serve/key-concepts.html#deployment), each optimized for its workload:
```
┌─────────────────────┐
┌───▶│ VideoEncoder │
│ │ (GPU) │
│ ├─────────────────────┤
┌─────────────────────┐ │ │ • SigLIP embedding │
│ VideoAnalyzer │──┤ │ • 16 frames/chunk │
│ (Ingress) │ │ │ • L2 normalization │
├─────────────────────┤ │ └─────────────────────┘
│ • S3 download │ │ num_gpus=1
│ • FFmpeg chunking │ │
│ • Request routing │ │ ┌─────────────────────┐
└─────────────────────┘ └───▶│ MultiDecoder │
num_cpus=6 │ (CPU) │
├─────────────────────┤
│ • Tag classification│
│ • Caption retrieval │
│ • Scene detection │
└─────────────────────┘
num_cpus=1
```
**Request lifecycle:**
1. `VideoAnalyzer` receives HTTP request with S3 video URI
2. Downloads video from S3, splits into fixed-duration chunks using FFmpeg
3. Sends all chunks to `VideoEncoder` concurrently
4. Encoder returns embedding references (stored in Ray object store)
5. `VideoAnalyzer` sends embeddings to `MultiDecoder` serially (for EMA state continuity)
6. Aggregates results and returns tags, captions, and scene changes
## Why Ray Serve?
This pipeline has three distinct workloads: a GPU-bound encoder running SigLIP, a CPU-bound decoder doing cosine similarity, and a CPU-heavy ingress running FFmpeg. Traditional serving frameworks force you to bundle these into a single container with fixed resources, wasting GPU when the decoder runs or starving FFmpeg when the encoder dominates.
Ray Serve solves this with **heterogeneous [resource allocation](https://docs.ray.io/en/latest/serve/configure-serve-deployment.html#configure-ray-serve-deployments) per deployment**. The encoder requests 1 GPU, the decoder requests 1 CPU, and the ingress requests 6 CPUs for parallel FFmpeg. Each deployment [scales independently](https://docs.ray.io/en/latest/serve/autoscaling-guide.html) based on its own queue depth—GPU replicas scale with encoding demand while CPU replicas scale separately with decoding demand. The load test demonstrates this: throughput scales near-linearly from 2.4 to 67.5 requests/second as the system provisions replicas to match load.
The pipeline also benefits from **zero-copy data transfer**. The ingress passes encoder results directly to the decoder as unawaited [DeploymentResponse](https://docs.ray.io/en/latest/serve/api/doc/ray.serve.handle.DeploymentResponse.html) references rather than serialized data. Ray stores the embeddings in its [object store](https://docs.ray.io/en/latest/ray-core/objects.html), and the decoder retrieves them directly without routing through the ingress. When encoder and decoder land on the same node, this transfer is zero-copy.
**Request pipelining** keeps the GPU saturated. By allowing two concurrent requests per encoder replica via `max_ongoing_requests`, one request prepares data on CPU while another computes on GPU. This achieves 100% GPU utilization without batching, which would add latency from waiting for requests to accumulate.
Finally, **[deployment composition](https://docs.ray.io/en/latest/serve/model_composition.html)** lets you define the encoder, decoder, and ingress as separate classes, then wire them together with `.bind()`. Ray Serve handles deployment ordering, health checks, and request routing. The ingress maintains explicit state (EMA for scene detection) across chunks, which works correctly even when autoscaling routes requests to different decoder replicas—no sticky sessions required.
---
## Setup
### Prerequisites
Before running this notebook, ensure you have:
| Requirement | Purpose |
|-------------|---------|
| **Pexels API key** | Download sample video (free at https://www.pexels.com/api/) |
| **S3 bucket** | Store videos and text embeddings |
| **AWS credentials** | Read/write access to your S3 bucket |
| **ffmpeg** | Video processing and frame extraction |
| **GPU** | Run SigLIP model for encoding (1 GPU minimum) |
Set these environment variables before running:
```bash
export PEXELS_API_KEY="your-pexels-api-key"
export S3_BUCKET="your-bucket-name"
export AWS_ACCESS_KEY_ID="..."
export AWS_SECRET_ACCESS_KEY="..."
```
> **Note on GPU type**: The benchmarks, design choices, and hyperparameters in this notebook were tuned for **NVIDIA L4 GPUs**. Different GPU types (A10G, T4, A100, etc.) have different memory bandwidth, compute throughput, and batch characteristics. You may need to adjust `max_ongoing_requests`, chunk sizes, and concurrency limits for optimal performance on other hardware.
```python
import os
PEXELS_API_KEY = os.environ.get("PEXELS_API_KEY") # Or set directly: "your-api-key"
S3_BUCKET = os.environ.get("S3_BUCKET") # Or set directly: "your-bucket"
```
### Download sample video
Before running the pipeline, we need a sample video in S3. This section downloads a video from Pexels, normalizes it, and uploads to S3.
**Why normalize videos?** We re-encode all videos to a consistent format:
- **384×384 resolution**: Matches SigLIP's input size exactly, eliminating resize during inference
- **30 fps**: Standardizes frame timing for consistent chunk boundaries
- **[H.264](https://en.wikipedia.org/wiki/Advanced_Video_Coding) codec (libx264)**: Fast seeking—FFmpeg can jump directly to any timestamp without decoding preceding frames. Some source codecs (VP9, HEVC) require sequential decoding, adding latency for chunk extraction
```python
import asyncio
try:
asyncio.get_running_loop()
except RuntimeError:
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
```
```python
from scripts.download_stock_videos import download_sample_videos
# Download sample videos (checks for existing manifest first, skips Pexels API if found)
S3_PREFIX = "anyscale-example/stock-videos/"
video_paths = asyncio.get_event_loop().run_until_complete(download_sample_videos(
api_key=PEXELS_API_KEY,
bucket=S3_BUCKET,
total=1, # Just need one sample video
s3_prefix=S3_PREFIX,
overwrite=False
))
if not video_paths:
raise RuntimeError("No videos downloaded")
SAMPLE_VIDEO_URI = video_paths[0]
print(f"\nSample video ready: {SAMPLE_VIDEO_URI}")
```
✅ S3 bucket 'anyscale-example-video-analysis-test-bucket' accessible ✅ Found existing manifest with 1 videos in S3 Skipping Pexels API download
Sample video ready: s3://anyscale-example-video-analysis-test-bucket/anyscale-example/stock-videos/kitchen_cooking_35395675_00.mp4
### Generate embeddings for text bank
The decoder matches video embeddings against precomputed **text embeddings** for tags and descriptions. We define the text banks here and use a [Ray task](https://docs.ray.io/en/latest/ray-core/tasks.html) to compute embeddings on GPU and upload to S3.
```python
from textbanks import TAGS, DESCRIPTIONS
print(f"Tags: {TAGS}")
print(f"Descriptions: {DESCRIPTIONS}")
```
Tags: ['kitchen', 'living room', 'office', 'meeting room', 'classroom', 'restaurant', 'cafe', 'grocery store', 'gym', 'warehouse', 'parking lot', 'city street', 'park', 'shopping mall', 'beach', 'sports field', 'hallway', 'lobby', 'bathroom', 'bedroom'] Descriptions: ['A person cooking in a kitchen', 'Someone preparing food on a counter', 'A chef working in a professional kitchen', 'People eating at a dining table', 'A group having a meal together', 'A person working at a desk', 'Someone typing on a laptop', 'A business meeting in progress', 'A presentation being given', 'People collaborating in an office', 'A teacher lecturing in a classroom', 'Students sitting at desks', 'A person giving a speech', 'Someone writing on a whiteboard', 'A customer shopping in a store', 'People browsing products on shelves', 'A cashier at a checkout counter', 'A person exercising at a gym', 'Someone lifting weights', 'A person running on a treadmill', 'People walking on a city sidewalk', 'Pedestrians crossing a street', 'Traffic moving through an intersection', 'Cars driving on a road', 'A vehicle parked in a lot', 'People walking through a park', 'Someone jogging outdoors', 'A group having a conversation', 'Two people talking face to face', 'A person on a phone call', 'Someone reading a book', 'A person watching television', 'People waiting in line', 'A crowded public space', 'An empty hallway or corridor', 'A person entering a building', 'Someone opening a door', 'A delivery being made', 'A person carrying boxes', 'Workers in a warehouse']
```python
import ray
from jobs.generate_text_embeddings import generate_embeddings_task
S3_EMBEDDINGS_PREFIX = "anyscale-example/embeddings/"
# Run the Ray task (uses TAGS and DESCRIPTIONS from textbanks module)
# Note: runtime_env ships local modules to worker nodes (job working_dir only applies to driver)
ray.init(runtime_env={"working_dir": "."}, ignore_reinit_error=True)
result = ray.get(generate_embeddings_task.remote(S3_BUCKET, S3_EMBEDDINGS_PREFIX))
print(f"\nText embeddings ready:")
print(f" Tags: {result['tag_embeddings']['s3_uri']}")
print(f" Descriptions: {result['description_embeddings']['s3_uri']}")
```
2026-01-06 08:27:08,101 INFO worker.py:1821 -- Connecting to existing Ray cluster at address: 10.0.45.10:6379... 2026-01-06 08:27:08,114 INFO worker.py:1998 -- Connected to Ray cluster. View the dashboard at https://session-lqu9h8iu3cpgv59j74p498djis.i.anyscaleuserdata-staging.com  2026-01-06 08:27:08,160 INFO packaging.py:463 -- Pushing file package 'gcs://_ray_pkg_56a868c6743600cdc03ad7fedef93ccaf9011e05.zip' (10.59MiB) to Ray cluster... 2026-01-06 08:27:08,200 INFO packaging.py:476 -- Successfully pushed file package 'gcs://_ray_pkg_56a868c6743600cdc03ad7fedef93ccaf9011e05.zip'. /home/ray/anaconda3/lib/python3.12/site-packages/ray/_private/worker.py:2046: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0 warnings.warn(
(generate_embeddings_task pid=6626, ip=10.0.222.50) ============================================================ (generate_embeddings_task pid=6626, ip=10.0.222.50) Starting text embedding generation (generate_embeddings_task pid=6626, ip=10.0.222.50) ============================================================ (generate_embeddings_task pid=6626, ip=10.0.222.50) (generate_embeddings_task pid=6626, ip=10.0.222.50) 📚 Loading text banks... (generate_embeddings_task pid=6626, ip=10.0.222.50) Tags: 20 (generate_embeddings_task pid=6626, ip=10.0.222.50) Descriptions: 40 (generate_embeddings_task pid=6626, ip=10.0.222.50) (generate_embeddings_task pid=6626, ip=10.0.222.50) 🤖 Loading SigLIP model: google/siglip-so400m-patch14-384 (generate_embeddings_task pid=6626, ip=10.0.222.50) Device: cuda
(generate_embeddings_task pid=6626, ip=10.0.222.50) Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
(generate_embeddings_task pid=6626, ip=10.0.222.50) Model loaded in 2.6s (generate_embeddings_task pid=6626, ip=10.0.222.50) (generate_embeddings_task pid=6626, ip=10.0.222.50) 🏷️ Generating tag embeddings... (generate_embeddings_task pid=6626, ip=10.0.222.50) Shape: (20, 1152) (generate_embeddings_task pid=6626, ip=10.0.222.50) Time: 0.21s (generate_embeddings_task pid=6626, ip=10.0.222.50) (generate_embeddings_task pid=6626, ip=10.0.222.50) 📝 Generating description embeddings... (generate_embeddings_task pid=6626, ip=10.0.222.50) Shape: (40, 1152) (generate_embeddings_task pid=6626, ip=10.0.222.50) Time: 0.25s (generate_embeddings_task pid=6626, ip=10.0.222.50) (generate_embeddings_task pid=6626, ip=10.0.222.50) ☁️ Uploading to S3 bucket: anyscale-example-video-analysis-test-bucket
Text embeddings ready: Tags: s3://anyscale-example-video-analysis-test-bucket/anyscale-example/embeddings/tag_embeddings.npz Descriptions: s3://anyscale-example-video-analysis-test-bucket/anyscale-example/embeddings/description_embeddings.npz
---
## Build the Ray Serve application
This section walks through building the video analysis pipeline step by step, introducing Ray Serve concepts as we go.
### GPU encoder
The `VideoEncoder` deployment runs on GPU and converts video frames to embeddings using SigLIP. Key configuration:
- `num_gpus=1`: Each replica requires a dedicated GPU
- `max_ongoing_requests=2`: Allows pipelining—while one request computes on GPU, another prepares data on CPU
**Why no request batching?** A single chunk (16 frames @ 384×384) already saturates GPU compute. Batching multiple requests would require holding them until a batch forms, adding latency without throughput gain. Instead, we use `max_ongoing_requests=2` to pipeline preparation and computation.
![GPU Utilization](assets/gpu_utilization.png)
**Why [`asyncio.to_thread`](https://docs.python.org/3/library/asyncio-task.html#asyncio.to_thread)?** Ray Serve deployments run in an async event loop. The `encode_frames` method is CPU/GPU-bound (PyTorch inference), which would block the event loop and prevent concurrent request handling. Wrapping it in `asyncio.to_thread` offloads the blocking work to a thread pool, keeping the event loop free to accept new requests.
**Why pass [`DeploymentResponse`](https://docs.ray.io/en/latest/serve/api/doc/ray.serve.handle.DeploymentResponse.html) to decoder?** Instead of awaiting the encoder result in `VideoAnalyzer` and passing raw data to the decoder, we pass the unawaited `DeploymentResponse` directly. Ray Serve automatically resolves this reference when the decoder needs it, storing the embeddings in the [object store](https://docs.ray.io/en/latest/ray-core/objects.html#objects-in-ray). This avoids an unnecessary serialize/deserialize round-trip through `VideoAnalyzer`—the decoder retrieves data directly from the object store, enabling zero-copy transfer if encoder and decoder are on the same node.
```python
from constants import MODEL_NAME
print(f"MODEL_NAME: {MODEL_NAME}")
```
(generate_embeddings_task pid=6626, ip=10.0.222.50) Tags: s3://anyscale-example-video-analysis-test-bucket/anyscale-example/embeddings/tag_embeddings.npz (generate_embeddings_task pid=6626, ip=10.0.222.50) Descriptions: s3://anyscale-example-video-analysis-test-bucket/anyscale-example/embeddings/description_embeddings.npz (generate_embeddings_task pid=6626, ip=10.0.222.50) (generate_embeddings_task pid=6626, ip=10.0.222.50) ✅ Done!
MODEL_NAME: google/siglip-so400m-patch14-384
```python
from deployments.encoder import VideoEncoder
import inspect
print(inspect.getsource(VideoEncoder.func_or_class))
```
@serve.deployment( num_replicas="auto", ray_actor_options={"num_gpus": 1, "num_cpus": 2}, # GPU utilization is at 100% when this is set to 2. with L4 # aka number on ongoing chunks that can be processed at once. max_ongoing_requests=2, autoscaling_config={ "min_replicas": 1, "max_replicas": 10, "target_num_ongoing_requests": 2, }, ) class VideoEncoder: """ Encodes video frames into embeddings using SigLIP.
Returns both per-frame embeddings and pooled embedding. """
def __init__(self): self.device = "cuda" if torch.cuda.is_available() else "cpu" print(f"VideoEncoder initializing on {self.device}")
# Load SigLIP model and processor self.processor = AutoProcessor.from_pretrained(MODEL_NAME) self.model = AutoModel.from_pretrained(MODEL_NAME).to(self.device) self.model.eval()
# Get embedding dimension self.embedding_dim = self.model.config.vision_config.hidden_size
print(f"VideoEncoder ready (embedding_dim={self.embedding_dim})")
def encode_frames(self, frames: np.ndarray) -> np.ndarray: """ Encode frames and return per-frame embeddings.
Args: frames: np.ndarray of shape (T, H, W, 3) uint8 RGB
Returns: np.ndarray of shape (T, D) float32, L2-normalized per-frame embeddings """
# Convert to PIL images pil_images = frames_to_pil_list(frames)
# Process images inputs = self.processor(images=pil_images, return_tensors="pt").to(self.device) # inputs = {k: v.to(self.device) for k, v in inputs.items()}
start = torch.cuda.Event(enable_timing=True) end = torch.cuda.Event(enable_timing=True)
# Get embeddings with torch.no_grad(): with torch.amp.autocast(device_type=self.device, enabled=self.device == "cuda"): outputs = self.model.get_image_features(**inputs)
# L2 normalize on GPU (faster than CPU numpy) frame_embeddings = torch.nn.functional.normalize(outputs, p=2, dim=1)
# Move to CPU and convert to numpy result = frame_embeddings.cpu().numpy().astype(np.float32) return result
async def encode_unbatched(self, frames: np.ndarray) -> dict: """ Unbatched entry point - processes single request directly.
Args: frames: np.ndarray of shape (T, H, W, 3)
Returns: dict with 'frame_embeddings' and 'embedding_dim' """ print(f"Unbatched: {frames.shape[0]} frames")
frame_embeddings = await asyncio.to_thread(self.encode_frames, frames)
return { "frame_embeddings": frame_embeddings, "embedding_dim": self.embedding_dim, }
@serve.batch(max_batch_size=2, batch_wait_timeout_s=0.1) async def encode_batched(self, frames_batch: List[np.ndarray]) -> List[dict]: """ Batched entry point - collects multiple requests into single GPU call.
Args: frames_batch: List of frame arrays, each of shape (T, H, W, 3)
Returns: List of dicts, each with 'frame_embeddings' and 'embedding_dim' """ frame_counts = [f.shape[0] for f in frames_batch] total_frames = sum(frame_counts)
print(f"Batched: {len(frames_batch)} requests ({total_frames} total frames)")
# Concatenate all frames into single batch all_frames = np.concatenate(frames_batch, axis=0)
# Single forward pass for all frames all_embeddings = await asyncio.to_thread(self.encode_frames, all_frames)
# Split results back per request results = [] offset = 0 for n_frames in frame_counts: chunk_embeddings = all_embeddings[offset:offset + n_frames] results.append({ "frame_embeddings": chunk_embeddings, "embedding_dim": self.embedding_dim, }) offset += n_frames
return results
async def __call__(self, frames: np.ndarray, use_batching: bool = False) -> dict: """ Main entry point. Set use_batching=False for direct comparison. """ if use_batching: return await self.encode_batched(frames) else: return await self.encode_unbatched(frames)
### CPU decoder
The `MultiDecoder` deployment runs on CPU and performs three tasks:
1. **Tag classification**: [Cosine similarity](https://en.wikipedia.org/wiki/Cosine_similarity) between video embedding and precomputed tag embeddings
2. **Caption retrieval**: Find the best-matching description from a text bank
3. **Scene detection**: EMA-based anomaly detection comparing each frame to recent history
The decoder loads precomputed text embeddings from S3 at startup.
**Why separate GPU/CPU deployments?** The encoder needs GPU for neural network inference; the decoder only does numpy dot products. Separating them allows independent scaling—encoders are limited by GPU count, decoders scale cheaply with CPU cores. This avoids tying expensive GPU resources to lightweight CPU work.
**Why EMA for scene detection?** Exponential Moving Average reuses existing SigLIP embeddings without an extra model. The algorithm computes `score = 1 - cosine(frame, EMA)` where EMA updates as `ema = 0.9*ema + 0.1*frame`. A simple threshold (`score > 0.15`) detects abrupt scene changes while smoothing noise.
```python
from deployments.decoder import MultiDecoder
print(inspect.getsource(MultiDecoder.func_or_class))
```
@serve.deployment( num_replicas="auto", ray_actor_options={"num_cpus": 1}, max_ongoing_requests=4, # can be set higher than 4, but since the encoder is limited to 4, we need to keep it at 4. autoscaling_config={ "min_replicas": 1, "max_replicas": 10, "target_num_ongoing_requests": 2, }, ) class MultiDecoder: """ Decodes video embeddings into tags, captions, and scene changes.
Uses precomputed text embeddings loaded from S3. This deployment is stateless - EMA state for scene detection is passed in and returned with each call, allowing the caller to maintain state continuity across multiple replicas. """
async def __init__(self, bucket: str, s3_prefix: str = S3_EMBEDDINGS_PREFIX): """Initialize decoder with text embeddings from S3.""" self.bucket = bucket self.ema_alpha = EMA_ALPHA self.scene_threshold = SCENE_CHANGE_THRESHOLD self.s3_prefix = s3_prefix logger.info(f"MultiDecoder initializing (bucket={self.bucket}, ema_alpha={self.ema_alpha}, threshold={self.scene_threshold})")
await self._load_embeddings()
logger.info(f"MultiDecoder ready (tags={len(self.tag_texts)}, descriptions={len(self.desc_texts)})")
async def _load_embeddings(self): """Load precomputed text embeddings from S3.""" session = aioboto3.Session(region_name=get_s3_region(self.bucket))
async with session.client("s3") as s3: # Load tag embeddings tag_key = f"{self.s3_prefix}tag_embeddings.npz" response = await s3.get_object(Bucket=self.bucket, Key=tag_key) tag_data = await response["Body"].read() tag_npz = np.load(io.BytesIO(tag_data), allow_pickle=True) self.tag_embeddings = tag_npz["embeddings"] self.tag_texts = tag_npz["texts"].tolist()
# Load description embeddings desc_key = f"{self.s3_prefix}description_embeddings.npz" response = await s3.get_object(Bucket=self.bucket, Key=desc_key) desc_data = await response["Body"].read() desc_npz = np.load(io.BytesIO(desc_data), allow_pickle=True) self.desc_embeddings = desc_npz["embeddings"] self.desc_texts = desc_npz["texts"].tolist()
def _cosine_similarity(self, embedding: np.ndarray, bank: np.ndarray) -> np.ndarray: """Compute cosine similarity between embedding and all vectors in bank.""" return bank @ embedding
def _get_top_tags(self, embedding: np.ndarray, top_k: int = 5) -> list[dict]: """Get top-k matching tags with scores.""" scores = self._cosine_similarity(embedding, self.tag_embeddings) top_indices = np.argsort(scores)[::-1][:top_k] return [ {"text": self.tag_texts[i], "score": float(scores[i])} for i in top_indices ]
def _get_retrieval_caption(self, embedding: np.ndarray) -> dict: """Get best matching description.""" scores = self._cosine_similarity(embedding, self.desc_embeddings) best_idx = np.argmax(scores) return { "text": self.desc_texts[best_idx], "score": float(scores[best_idx]), }
def _detect_scene_changes( self, frame_embeddings: np.ndarray, chunk_index: int, chunk_start_time: float, chunk_duration: float, ema_state: np.ndarray | None = None, ) -> tuple[list[dict], np.ndarray]: """ Detect scene changes using EMA-based scoring.
score_t = 1 - cosine(E_t, ema_t) ema_t = α * ema_{t-1} + (1-α) * E_t
Args: frame_embeddings: (T, D) normalized embeddings chunk_index: Index of this chunk in the video chunk_start_time: Start time of chunk in video (seconds) chunk_duration: Duration of chunk (seconds) ema_state: EMA state from previous chunk, or None for first chunk
Returns: Tuple of (scene_changes list, updated ema_state) """ num_frames = len(frame_embeddings) if num_frames == 0: # Return empty changes and unchanged state (or zeros if no state) return [], ema_state if ema_state is not None else np.zeros(0)
# Initialize EMA from first frame if no prior state ema = ema_state.copy() if ema_state is not None else frame_embeddings[0].copy() scene_changes = []
for frame_idx, embedding in enumerate(frame_embeddings): # Compute score: how different is current frame from recent history similarity = float(np.dot(embedding, ema)) score = max(0.0, 1.0 - similarity)
# Detect scene change if score exceeds threshold if score >= self.scene_threshold: # Calculate timestamp within video frame_offset = (frame_idx / max(1, num_frames - 1)) * chunk_duration timestamp = chunk_start_time + frame_offset
scene_changes.append({ "timestamp": round(timestamp, 3), "score": round(score, 4), "chunk_index": chunk_index, "frame_index": frame_idx, })
# Update EMA ema = self.ema_alpha * ema + (1 - self.ema_alpha) * embedding # Re-normalize ema = ema / np.linalg.norm(ema)
return scene_changes, ema
def __call__( self, encoder_output: dict, chunk_index: int, chunk_start_time: float, chunk_duration: float, top_k_tags: int = 5, ema_state: np.ndarray | None = None, ) -> dict: """ Decode embeddings into tags, caption, and scene changes.
Args: encoder_output: Dict with 'frame_embeddings' and 'embedding_dim' chunk_index: Index of this chunk in the video chunk_start_time: Start time of chunk (seconds) chunk_duration: Duration of chunk (seconds) top_k_tags: Number of top tags to return ema_state: EMA state from previous chunk for scene detection continuity. Pass None for the first chunk of a stream.
Returns: Dict containing tags, retrieval_caption, scene_changes, and updated ema_state. The caller should pass the returned ema_state to the next chunk's call. """ # Get frame embeddings from encoder output frame_embeddings = encoder_output["frame_embeddings"]
# Calculate pooled embedding (mean across frames, normalized) pooled_embedding = frame_embeddings.mean(axis=0) pooled_embedding = pooled_embedding / np.linalg.norm(pooled_embedding)
# Classification and retrieval on pooled embedding tags = self._get_top_tags(pooled_embedding, top_k=top_k_tags) caption = self._get_retrieval_caption(pooled_embedding)
# Scene change detection on frame embeddings scene_changes, new_ema_state = self._detect_scene_changes( frame_embeddings=frame_embeddings, chunk_index=chunk_index, chunk_start_time=chunk_start_time, chunk_duration=chunk_duration, ema_state=ema_state, )
return { "tags": tags, "retrieval_caption": caption, "scene_changes": scene_changes, "ema_state": new_ema_state, }
### Video chunking
Before we can process a video, we need to split it into fixed-duration chunks and extract frames. The chunking process:
1. **Get video duration** using [`ffprobe`](https://ffmpeg.org/ffprobe.html)
2. **Define chunk boundaries** (e.g., 0-10s, 10-20s, 20-30s for a 30s video)
3. **Extract frames in parallel** using multiple concurrent [`ffmpeg`](https://ffmpeg.org/) processes
4. **Limit concurrency** with [`asyncio.Semaphore`](https://docs.python.org/3/library/asyncio-sync.html#asyncio.Semaphore) to avoid CPU oversubscription
Each chunk extracts 16 frames uniformly sampled across its duration, resized to 384×384 (matching SigLIP's input size).
#### Design choices
**Direct S3 download vs presigned URLs**: We download the video to local disk before processing. An alternative is generating a presigned S3 URL and passing it directly to FFmpeg. Benchmarks show direct download is faster—FFmpeg's HTTP client doesn't handle S3's chunked responses as efficiently as `aioboto3`, and network latency compounds across multiple seeks.
![Presigned vs Direct S3](assets/presigned_vs_direct_s3.png)
**Single FFmpeg vs parallel FFmpeg**: Two approaches for extracting frames from multiple chunks:
- **Single FFmpeg**: One process reads the entire video, using `select` filter to pick frames at specific timestamps
- **Parallel FFmpeg**: Multiple concurrent processes, each extracting one chunk
Chose the Single FFmpeg approach since it outperforms parallel FFmpeg on longer videos and yields similar performance for typical 10s chunks. This method is both efficient and scalable as chunk counts grow.
![Single vs Multi FFmpeg](assets/single_vs_multi_ffmpeg.png)
**Chunk duration**: We use 10-second chunks. Shorter chunks increase overhead (more FFmpeg calls, more encoder/decoder round-trips). Longer chunks increases processing efficiency but **degrade inference quality**—SigLIP processes exactly 16 frames per chunk, so a 60-second chunk samples one frame every 3.75 seconds, missing fast scene changes. The 10-second sweet spot balances throughput with temporal resolution (~1.6 fps sampling).
![Chunk Video Analysis](assets/chunk_video_analysis.png)
### Deployment composition
The `VideoAnalyzer` ingress deployment orchestrates the encoder and decoder. It uses [FastAPI](https://fastapi.tiangolo.com/) integration with [`@serve.ingress`](https://docs.ray.io/en/latest/serve/http-guide.html#fastapi-http-deployments) for HTTP endpoints.
#### Design choices
**Why `num_cpus=6`?** The analyzer runs FFmpeg for frame extraction. Each FFmpeg process uses `FFMPEG_THREADS=2`, and we run up to `NUM_WORKERS=3` concurrent processes. So `2 × 3 = 6` CPUs ensures FFmpeg doesn't contend for CPU during parallel chunk extraction.
**Why `max_ongoing_requests=4`?** The encoder has `max_ongoing_requests=2`. We want the analyzer to stay ahead: while 2 videos are encoding, we download and chunk 2 more videos so they're ready when encoder slots free up. This keeps the GPU pipeline saturated without excessive memory from queued requests.
**Why cache the S3 client?** Creating a new `aioboto3` client per request adds overhead (connection setup, credential resolution). Caching the client in `__init__` and reusing it across requests amortizes this cost. The client is thread-safe and handles connection pooling internally.
**Why encode in parallel but decode serially?** Encoding is stateless—each chunk's frames go through SigLIP independently, so we fire all chunks concurrently with [`asyncio.gather`](https://docs.python.org/3/library/asyncio-task.html#asyncio.gather). Decoding requires temporal ordering—the EMA for scene detection must process chunks in order (chunk 0's EMA state feeds into chunk 1). The `VideoAnalyzer` calls the decoder serially, passing EMA state from each response to the next request. This explicit state passing ensures correct scene detection even when multiple decoder replicas exist under autoscaling.
```python
from app import VideoAnalyzer
print(inspect.getsource(VideoAnalyzer.func_or_class))
```
@serve.deployment( # setting this to twice that of the encoder. So that requests can complete the # upfront CPU work and be queued for GPU processing. num_replicas="auto", ray_actor_options={"num_cpus": FFMPEG_THREADS}, max_ongoing_requests=4, autoscaling_config={ "min_replicas": 2, "max_replicas": 20, "target_num_ongoing_requests": 2, }, ) @serve.ingress(fastapi_app) class VideoAnalyzer: """ Main ingress deployment that orchestrates VideoEncoder and MultiDecoder.
Encoder refs are passed directly to decoder; Ray Serve resolves dependencies. Downloads video from S3 to temp file for fast local processing. """
def __init__(self, encoder: VideoEncoder, decoder: MultiDecoder): self.encoder = encoder self.decoder = decoder self._s3_session = aioboto3.Session() self._s3_client = None # Cached client for reuse across requests logger.info("VideoAnalyzer ready")
async def _get_s3_client(self): """Get or create a reusable S3 client.""" if self._s3_client is None: self._s3_client = await self._s3_session.client("s3").__aenter__() return self._s3_client
async def _download_video(self, s3_uri: str) -> Path: """Download video from S3 to temp file. Returns local path.""" bucket, key = parse_s3_uri(s3_uri)
# Create temp file with video extension suffix = Path(key).suffix or ".mp4" temp_file = tempfile.NamedTemporaryFile(suffix=suffix, delete=False) temp_path = Path(temp_file.name) temp_file.close()
s3 = await self._get_s3_client() await s3.download_file(bucket, key, str(temp_path))
return temp_path
def _aggregate_results( self, chunk_results: list[dict], top_k_tags: int = 5, ) -> dict: """ Aggregate results from multiple chunks.
Strategy:
- Tags: Average scores across chunks, return top-k
- Caption: Return the one with highest score across all chunks """ # Aggregate tag scores tag_scores = defaultdict(list) for result in chunk_results: for tag in result["tags"]: tag_scores[tag["text"]].append(tag["score"])
# Average tag scores and sort aggregated_tags = [ {"text": text, "score": np.mean(scores)} for text, scores in tag_scores.items() ] aggregated_tags.sort(key=lambda x: x["score"], reverse=True) top_tags = aggregated_tags[:top_k_tags]
# Best caption across all chunks best_caption = max( (r["retrieval_caption"] for r in chunk_results), key=lambda x: x["score"], )
return { "tags": top_tags, "retrieval_caption": best_caption, }
def _encode_chunk(self, frames: np.ndarray, use_batching: bool = False) -> DeploymentResponse: """Encode a single chunk's frames to embeddings. Returns DeploymentResponse ref.""" return self.encoder.remote(frames, use_batching=use_batching)
async def _decode_chunk( self, encoder_output: dict, chunk_index: int, chunk_start_time: float, chunk_duration: float, ema_state=None, ) -> dict: """Decode embeddings to tags, caption, scene changes.""" return await self.decoder.remote( encoder_output=encoder_output, chunk_index=chunk_index, chunk_start_time=chunk_start_time, chunk_duration=chunk_duration, ema_state=ema_state, )
@fastapi_app.post("/analyze", response_model=AnalyzeResponse) async def analyze(self, request: AnalyzeRequest) -> AnalyzeResponse: """ Analyze a video from S3 and return tags, caption, and scene changes.
Downloads video to temp file for fast local processing. Chunks the entire video and aggregates results. Encoder refs are passed directly to decoder for dependency resolution. """ total_start = time.perf_counter() temp_path = None
try: # Download video from S3 to temp file download_start = time.perf_counter() try: temp_path = await self._download_video(request.video_path) except Exception as e: raise HTTPException(status_code=400, detail=f"Cannot download S3 video: {e}") s3_download_ms = (time.perf_counter() - download_start) * 1000
# Chunk video with PARALLEL frame extraction from local file decode_start = time.perf_counter() try: chunks = await chunk_video_async( str(temp_path), chunk_duration=request.chunk_duration, num_frames_per_chunk=request.num_frames, ffmpeg_threads=FFMPEG_THREADS, use_single_ffmpeg=True, ) except Exception as e: raise HTTPException(status_code=400, detail=f"Cannot process video: {e}")
decode_video_ms = (time.perf_counter() - decode_start) * 1000
if not chunks: raise HTTPException(status_code=400, detail="No chunks extracted from video")
# Calculate video duration from chunks video_duration = chunks[-1].start_time + chunks[-1].duration
# Fire off all encoder calls (returns refs, not awaited) encode_start = time.perf_counter() encode_refs = [ self._encode_chunk(chunk.frames, use_batching=request.use_batching) for chunk in chunks ] encode_ms = (time.perf_counter() - encode_start) * 1000
# Decode chunks SERIALLY, passing encoder refs directly. # Ray Serve resolves the encoder result when decoder needs it. # EMA state is tracked here (not in decoder) to ensure continuity # even when autoscaling routes requests to different replicas. decode_start = time.perf_counter() decode_results = [] ema_state = None # Will be initialized from first chunk's first frame for chunk, enc_ref in zip(chunks, encode_refs): dec_result = await self._decode_chunk( encoder_output=enc_ref, chunk_index=chunk.index, chunk_start_time=chunk.start_time, chunk_duration=chunk.duration, ema_state=ema_state, ) decode_results.append(dec_result) ema_state = dec_result["ema_state"] # Carry forward for next chunk decode_ms = (time.perf_counter() - decode_start) * 1000
# Collect results chunk_results = [] per_chunk_results = [] all_scene_changes = []
for chunk, decoder_result in zip(chunks, decode_results): chunk_results.append(decoder_result)
# Scene changes come directly from decoder chunk_scene_changes = [ SceneChange(**sc) for sc in decoder_result["scene_changes"] ] all_scene_changes.extend(chunk_scene_changes)
per_chunk_results.append(ChunkResult( chunk_index=chunk.index, start_time=chunk.start_time, duration=chunk.duration, tags=[TagResult(**t) for t in decoder_result["tags"]], retrieval_caption=CaptionResult(**decoder_result["retrieval_caption"]), scene_changes=chunk_scene_changes, ))
# Aggregate results aggregated = self._aggregate_results(chunk_results)
total_ms = (time.perf_counter() - total_start) * 1000
return AnalyzeResponse( stream_id=request.stream_id, tags=[TagResult(**t) for t in aggregated["tags"]], retrieval_caption=CaptionResult(**aggregated["retrieval_caption"]), scene_changes=all_scene_changes, num_scene_changes=len(all_scene_changes), chunks=per_chunk_results, num_chunks=len(chunks), video_duration=video_duration, timing_ms=TimingResult( s3_download_ms=round(s3_download_ms, 2), decode_video_ms=round(decode_video_ms, 2), encode_ms=round(encode_ms, 2), decode_ms=round(decode_ms, 2), total_ms=round(total_ms, 2), ), ) finally: # Clean up temp file if temp_path and temp_path.exists(): temp_path.unlink(missing_ok=True)
@fastapi_app.get("/health") async def health(self): """Health check endpoint.""" return {"status": "healthy"}
```python
encoder = VideoEncoder.bind()
decoder = MultiDecoder.bind(bucket=S3_BUCKET, s3_prefix=S3_EMBEDDINGS_PREFIX)
app = VideoAnalyzer.bind(encoder=encoder, decoder=decoder)
```
```python
from ray import serve
serve.run(app, name="video-analyzer", route_prefix="/", blocking=False)
```
(ProxyActor pid=251250) INFO 2026-01-06 08:27:23,294 proxy 10.0.45.10 -- Proxy starting on node 7beb9ebc3808dac027f36a0400592d7a27cc9e90f0aab0a2578c328b (HTTP port: 8000). INFO 2026-01-06 08:27:23,369 serve 250708 -- Started Serve in namespace "serve". (ServeController pid=251181) INFO 2026-01-06 08:27:23,394 controller 251181 -- Registering autoscaling state for deployment Deployment(name='VideoEncoder', app='video-analyzer') (ServeController pid=251181) INFO 2026-01-06 08:27:23,395 controller 251181 -- Deploying new version of Deployment(name='VideoEncoder', app='video-analyzer') (initial target replicas: 1). (ServeController pid=251181) INFO 2026-01-06 08:27:23,397 controller 251181 -- Registering autoscaling state for deployment Deployment(name='MultiDecoder', app='video-analyzer') (ServeController pid=251181) INFO 2026-01-06 08:27:23,397 controller 251181 -- Deploying new version of Deployment(name='MultiDecoder', app='video-analyzer') (initial target replicas: 1). (ServeController pid=251181) INFO 2026-01-06 08:27:23,398 controller 251181 -- Registering autoscaling state for deployment Deployment(name='VideoAnalyzer', app='video-analyzer') (ServeController pid=251181) INFO 2026-01-06 08:27:23,398 controller 251181 -- Deploying new version of Deployment(name='VideoAnalyzer', app='video-analyzer') (initial target replicas: 2). (ProxyActor pid=251250) INFO 2026-01-06 08:27:23,365 proxy 10.0.45.10 -- Got updated endpoints: {}. (ProxyActor pid=251250) INFO 2026-01-06 08:27:23,403 proxy 10.0.45.10 -- Got updated endpoints: {Deployment(name='VideoAnalyzer', app='video-analyzer'): EndpointInfo(route='/', app_is_cross_language=False, route_patterns=None)}. (ServeController pid=251181) INFO 2026-01-06 08:27:23,508 controller 251181 -- Adding 1 replica to Deployment(name='VideoEncoder', app='video-analyzer'). (ServeController pid=251181) INFO 2026-01-06 08:27:23,510 controller 251181 -- Adding 1 replica to Deployment(name='MultiDecoder', app='video-analyzer'). (ServeController pid=251181) INFO 2026-01-06 08:27:23,512 controller 251181 -- Adding 2 replicas to Deployment(name='VideoAnalyzer', app='video-analyzer'). (ProxyActor pid=251250) INFO 2026-01-06 08:27:23,416 proxy 10.0.45.10 -- Started <ray.serve._private.router.SharedRouterLongPollClient object at 0x72d0e5f4f710>. (ProxyActor pid=251250) INFO 2026-01-06 08:27:27,576 proxy 10.0.45.10 -- Got updated endpoints: {Deployment(name='VideoAnalyzer', app='video-analyzer'): EndpointInfo(route='/', app_is_cross_language=False, route_patterns=[RoutePattern(methods=['POST'], path='/analyze'), RoutePattern(methods=['GET', 'HEAD'], path='/docs'), RoutePattern(methods=['GET', 'HEAD'], path='/docs/oauth2-redirect'), RoutePattern(methods=['GET'], path='/health'), RoutePattern(methods=['GET', 'HEAD'], path='/openapi.json'), RoutePattern(methods=['GET', 'HEAD'], path='/redoc')])}. (ProxyActor pid=67913, ip=10.0.239.104) INFO 2026-01-06 08:27:28,373 proxy 10.0.239.104 -- Proxy starting on node 6954457300d89edbf779f01ef0bfcfc5d109d927fca0e3b4f80974ba (HTTP port: 8000). [repeated 2x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.) (ProxyActor pid=6627, ip=10.0.222.50) INFO 2026-01-06 08:27:26,287 proxy 10.0.222.50 -- Got updated endpoints: {Deployment(name='VideoAnalyzer', app='video-analyzer'): EndpointInfo(route='/', app_is_cross_language=False, route_patterns=None)}.
(ServeReplica:video-analyzer:VideoEncoder pid=6736, ip=10.0.222.50) VideoEncoder initializing on cuda
(ServeReplica:video-analyzer:VideoEncoder pid=6736, ip=10.0.222.50) Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`. (ProxyActor pid=67913, ip=10.0.239.104) INFO 2026-01-06 08:27:28,444 proxy 10.0.239.104 -- Started <ray.serve._private.router.SharedRouterLongPollClient object at 0x7dd178a4d190>. [repeated 2x across cluster]
(ServeReplica:video-analyzer:VideoEncoder pid=6736, ip=10.0.222.50) VideoEncoder ready (embedding_dim=1152)
INFO 2026-01-06 08:27:31,518 serve 250708 -- Application 'video-analyzer' is ready at http://0.0.0.0:8000/. INFO 2026-01-06 08:27:31,534 serve 250708 -- Started <ray.serve._private.router.SharedRouterLongPollClient object at 0x78edac0dbef0>.
DeploymentHandle(deployment='VideoAnalyzer')
```python
import httpx
import time
# Wait for deployment to be ready
print("Waiting for deployment to be healthy...")
start = time.time()
while True:
try:
with httpx.Client(timeout=5.0) as client:
response = client.get("http://localhost:8000/health")
if response.status_code == 200:
print(f"Deployment ready in {time.time() - start:.1f}s")
break
except httpx.RequestError:
pass
time.sleep(1.0)
if time.time() - start > 120:
raise TimeoutError("Deployment did not become healthy within 120s")
```
Waiting for deployment to be healthy... Deployment ready in 0.0s
```python
import httpx
import time
import uuid
# Send a sample request to the deployed service
payload = {
"stream_id": uuid.uuid4().hex[:8],
"video_path": SAMPLE_VIDEO_URI,
"num_frames": 16,
"chunk_duration": 10.0,
}
print(f"Analyzing video: {SAMPLE_VIDEO_URI}")
print(f"Stream ID: {payload['stream_id']}\n")
start = time.perf_counter()
with httpx.Client(timeout=300.0) as client:
response = client.post("http://localhost:8000/analyze", json=payload)
latency_ms = (time.perf_counter() - start) * 1000
result = response.json()
# Print results
print("=" * 60)
print(f"Video duration: {result['video_duration']:.1f}s")
print(f"Chunks processed: {result['num_chunks']}")
print()
print("Top Tags:")
for tag in result["tags"]:
print(f" {tag['score']:.3f} {tag['text']}")
print()
print("Best Caption:")
print(f" {result['retrieval_caption']['score']:.3f} {result['retrieval_caption']['text']}")
print()
print(f"Scene Changes: {result['num_scene_changes']}")
for sc in result["scene_changes"][:5]: # Show first 5
print(f" {sc['timestamp']:6.2f}s score={sc['score']:.3f}")
print()
print("Timing:")
timing = result["timing_ms"]
print(f" S3 download: {timing['s3_download_ms']:7.1f} ms")
print(f" Video decode: {timing['decode_video_ms']:7.1f} ms")
print(f" Encode (GPU): {timing['encode_ms']:7.1f} ms")
print(f" Decode (CPU): {timing['decode_ms']:7.1f} ms")
print(f" Total server: {timing['total_ms']:7.1f} ms")
print(f" Round-trip: {latency_ms:7.1f} ms")
print("=" * 60)
```
(ServeReplica:video-analyzer:VideoAnalyzer pid=67729, ip=10.0.239.104) INFO 2026-01-06 08:27:31,893 video-analyzer_VideoAnalyzer npdr89ay b9cd7535-7acc-4fa4-a54b-a0e789ca9ce7 -- GET /health 200 1.5ms
Analyzing video: s3://anyscale-example-video-analysis-test-bucket/anyscale-example/stock-videos/kitchen_cooking_35395675_00.mp4 Stream ID: 766cc773
(ServeReplica:video-analyzer:VideoAnalyzer pid=67728, ip=10.0.239.104) INFO 2026-01-06 08:27:32,683 video-analyzer_VideoAnalyzer ntukkwkd 9326cc34-5b87-491c-8025-a7db9a2806f4 -- Started <ray.serve._private.router.SharedRouterLongPollClient object at 0x788bd83d2120>.
(ServeReplica:video-analyzer:VideoEncoder pid=6736, ip=10.0.222.50) Unbatched: 16 frames ============================================================ Video duration: 8.3s Chunks processed: 1
Top Tags: 0.082 kitchen 0.070 cafe 0.058 living room 0.057 bedroom 0.053 office
Best Caption: 0.101 Someone preparing food on a counter
Scene Changes: 0
Timing: S3 download: 238.8 ms Video decode: 100.6 ms Encode (GPU): 18.3 ms Decode (CPU): 855.0 ms Total server: 1212.9 ms Round-trip: 1235.5 ms ============================================================
(ServeReplica:video-analyzer:MultiDecoder pid=6737, ip=10.0.222.50) /home/ray/anaconda3/lib/python3.12/site-packages/ray/serve/_private/replica.py:1640: UserWarning: Calling sync method '__call__' directly on the asyncio loop. In a future version, sync methods will be run in a threadpool by default. Ensure your sync methods are thread safe or keep the existing behavior by making them `async def`. Opt into the new behavior by setting RAY_SERVE_RUN_SYNC_IN_THREADPOOL=1. (ServeReplica:video-analyzer:MultiDecoder pid=6737, ip=10.0.222.50) warnings.warn( (ServeReplica:video-analyzer:MultiDecoder pid=6737, ip=10.0.222.50) INFO 2026-01-06 08:27:33,539 video-analyzer_MultiDecoder 1carxuil 9326cc34-5b87-491c-8025-a7db9a2806f4 -- CALL __call__ OK 2.1ms (ServeReplica:video-analyzer:VideoEncoder pid=6736, ip=10.0.222.50) INFO 2026-01-06 08:27:33,533 video-analyzer_VideoEncoder pxjowb4x 9326cc34-5b87-491c-8025-a7db9a2806f4 -- CALL __call__ OK 804.7ms (ServeReplica:video-analyzer:VideoAnalyzer pid=67728, ip=10.0.239.104) INFO 2026-01-06 08:27:33,541 video-analyzer_VideoAnalyzer ntukkwkd 9326cc34-5b87-491c-8025-a7db9a2806f4 -- POST /analyze 200 1217.9ms
```python
serve.shutdown()
```
---
## Load Test Results
To evaluate the pipeline's performance under realistic conditions, we ran load tests using the `client/load_test.py` script with **concurrency levels ranging from 2 to 64 concurrent requests**. The Ray Serve application was configured with autoscaling enabled, allowing replicas to scale dynamically based on demand.
### Methodology
The load test was executed using:
```bash
python -m client.load_test --video s3://bucket/video.mp4 --concurrency <N>
```
**Test parameters:**
- Concurrency levels: 2, 4, 8, 16, 32, 64
- Chunk duration: 10 seconds
- Frames per chunk: 16
- Autoscaling: Enabled (replicas scale based on `target_num_ongoing_requests`)
The charts below show autoscaling in action during the concurrency=2 to 64 load test:
![Ongoing Requests](assets/ongoing.png)
![Replica Count](assets/num_replicas.png)
![Queue Length](assets/queue_len.png)
As load increases, replicas scale up to maintain target queue depth. The system reaches steady state once enough replicas are provisioned to handle the request rate.
To ensure fair comparison, we discarded the first half of each test run to exclude the autoscaling warm-up period where latencies are elevated due to replica initialization.
### Results Summary
| Concurrency | Requests | P50 (ms) | P95 (ms) | P99 (ms) | Throughput (req/s) |
|-------------|----------|----------|----------|----------|-------------------|
| 2 | 152 | 838 | 843 | 844 | 2.37 |
| 4 | 590 | 858 | 984 | 1,025 | 4.68 |
| 8 | 1,103 | 885 | 1,065 | 1,120 | 8.97 |
| 16 | 2,240 | 900 | 1,074 | 1,128 | 17.78 |
| 32 | 4,141 | 928 | 1,095 | 1,151 | 34.75 |
| 64 | 17,283 | 967 | 1,128 | 1,188 | 67.55 |
**Key findings:**
- **100% success rate** across all 25,509 requests analyzed
- **Near-linear throughput scaling**: Throughput increased from 2.37 req/s at concurrency 2 to **67.55 req/s at concurrency 64** (~28x improvement)
- **Stable latencies under load**: P95 latency remained between 843ms and 1,128ms across all concurrency levels, demonstrating effective autoscaling
### Processing Time Breakdown
The chart below shows how processing time is distributed across pipeline stages at each concurrency level:
![Processing Time Breakdown](assets/processing_time_breakdown.png)
**At concurrency 64 (best throughput):**
- **S3 Download**: 77ms (8%)
- **Video Decode (FFmpeg)**: 98ms (10%)
- **Encode (GPU)**: <1ms (<1%) — Note: This number is artificially low due to how timing is measured. Because the output of VideoEncoder is passed directly into MultiDecoder as an argument, the instrumentation mistakenly attributes most of the computational time to the downstream stage. In reality, VideoEncoder performs the bulk of the work, so its true processing time is significantly higher than reported here.
- **Decode (CPU)**: 757ms (81%)
The CPU decoding stage (tag/caption generation) dominates processing time. S3 download and video decoding are relatively fast due to local caching and optimized FFmpeg settings.
### Throughput Analysis
The charts below show throughput scaling and the latency-throughput tradeoff:
![Throughput Analysis](assets/throughput_analysis.png)
**Observations:**
1. **Linear scaling**: Throughput scales almost linearly with concurrency, indicating that autoscaling successfully provisions enough replicas to handle increased load.
2. **Latency-throughput tradeoff**: The right chart shows that P95 latency increases slightly as throughput grows (from ~843ms to ~1,128ms), but remains within acceptable bounds. This ~34% latency increase enables a ~28x throughput improvement.
3. **No saturation point**: Even at concurrency 64, throughput continues to scale. The system could likely handle higher concurrency with additional GPU resources.
The near-linear scaling demonstrates that the pipeline architecture—with separate GPU encoder, CPU decoder, and CPU-bound ingress—allows each component to scale independently based on its resource requirements.
doc/source/serve/tutorials/video-analysis/__init__.py
View File
doc/source/serve/tutorials/video-analysis/analyze_load_tests.ipynb
+1382
View File
File diff suppressed because one or more lines are too long
doc/source/serve/tutorials/video-analysis/app.py
+357
View File
@@ -0,0 +1,357 @@
"""
Ray Serve application: Video Embedding → Multi-Decoder.
Processes entire videos by chunking into segments.
Videos are downloaded from S3 to temp file, then processed locally (faster than streaming).
Encoder refs are passed directly to decoder; Ray Serve resolves dependencies automatically.
Usage:
serve run app:app
# With custom bucket:
S3_BUCKET=my-bucket serve run app:app
"""
import logging
import os
import tempfile
import time
from collections import defaultdict
from pathlib import Path
from urllib.parse import urlparse
import aioboto3
import numpy as np
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from ray import serve
from ray.serve.handle import DeploymentResponse
from deployments.encoder import VideoEncoder
from deployments.decoder import MultiDecoder
from utils.video import chunk_video_async
from constants import DEFAULT_NUM_FRAMES, DEFAULT_CHUNK_DURATION, FFMPEG_THREADS, NUM_WORKERS
logger = logging.getLogger(__name__)
def parse_s3_uri(s3_uri: str) -> tuple[str, str]:
"""Parse s3://bucket/key into (bucket, key)."""
parsed = urlparse(s3_uri)
if parsed.scheme != "s3":
raise ValueError(f"Invalid S3 URI: {s3_uri}")
bucket = parsed.netloc
key = parsed.path.lstrip("/")
return bucket, key
class AnalyzeRequest(BaseModel):
"""Request schema for /analyze endpoint."""
stream_id: str
video_path: str # S3 URI: s3://bucket/key
num_frames: int = DEFAULT_NUM_FRAMES
chunk_duration: float = DEFAULT_CHUNK_DURATION
use_batching: bool = False # Set False to compare unbatched performance
class TagResult(BaseModel):
text: str
score: float
class CaptionResult(BaseModel):
text: str
score: float
class TimingResult(BaseModel):
s3_download_ms: float
decode_video_ms: float
encode_ms: float
decode_ms: float
total_ms: float
class SceneChange(BaseModel):
"""Detected scene change event."""
timestamp: float # Seconds from video start
score: float # Scene change score (higher = bigger change)
chunk_index: int
frame_index: int # Frame index within chunk
class ChunkResult(BaseModel):
"""Result for a single chunk."""
chunk_index: int
start_time: float
duration: float
tags: list[TagResult]
retrieval_caption: CaptionResult
# Detected scene changes in this chunk
scene_changes: list[SceneChange]
class AnalyzeResponse(BaseModel):
"""Response schema for /analyze endpoint."""
stream_id: str
# Aggregated results (across all chunks)
tags: list[TagResult]
retrieval_caption: CaptionResult
# Scene change detection
scene_changes: list[SceneChange] # All detected scene changes
num_scene_changes: int
# Per-chunk results
chunks: list[ChunkResult]
num_chunks: int
video_duration: float
timing_ms: TimingResult
# FastAPI app
fastapi_app = FastAPI(
title="Video Embedding API",
description="GPU encoder → CPU multi-decoder using SigLIP embeddings",
)
@serve.deployment(
# setting this to twice that of the encoder. So that requests can complete the
# upfront CPU work and be queued for GPU processing.
num_replicas="auto",
ray_actor_options={"num_cpus": FFMPEG_THREADS},
max_ongoing_requests=4,
autoscaling_config={
"min_replicas": 2,
"max_replicas": 20,
"target_num_ongoing_requests": 2,
},
)
@serve.ingress(fastapi_app)
class VideoAnalyzer:
"""
Main ingress deployment that orchestrates VideoEncoder and MultiDecoder.
Encoder refs are passed directly to decoder; Ray Serve resolves dependencies.
Downloads video from S3 to temp file for fast local processing.
"""
def __init__(self, encoder: VideoEncoder, decoder: MultiDecoder):
self.encoder = encoder
self.decoder = decoder
self._s3_session = aioboto3.Session()
self._s3_client = None # Cached client for reuse across requests
logger.info("VideoAnalyzer ready")
async def _get_s3_client(self):
"""Get or create a reusable S3 client."""
if self._s3_client is None:
self._s3_client = await self._s3_session.client("s3").__aenter__()
return self._s3_client
async def _download_video(self, s3_uri: str) -> Path:
"""Download video from S3 to temp file. Returns local path."""
bucket, key = parse_s3_uri(s3_uri)
# Create temp file with video extension
suffix = Path(key).suffix or ".mp4"
temp_file = tempfile.NamedTemporaryFile(suffix=suffix, delete=False)
temp_path = Path(temp_file.name)
temp_file.close()
try:
s3 = await self._get_s3_client()
await s3.download_file(bucket, key, str(temp_path))
except Exception:
# Clean up temp file if download fails
temp_path.unlink(missing_ok=True)
raise
return temp_path
def _aggregate_results(
self,
chunk_results: list[dict],
top_k_tags: int = 5,
) -> dict:
"""
Aggregate results from multiple chunks.
Strategy:
- Tags: Average scores across chunks, return top-k
- Caption: Return the one with highest score across all chunks
"""
# Aggregate tag scores
tag_scores = defaultdict(list)
for result in chunk_results:
for tag in result["tags"]:
tag_scores[tag["text"]].append(tag["score"])
# Average tag scores and sort
aggregated_tags = [
{"text": text, "score": np.mean(scores)}
for text, scores in tag_scores.items()
]
aggregated_tags.sort(key=lambda x: x["score"], reverse=True)
top_tags = aggregated_tags[:top_k_tags]
# Best caption across all chunks
best_caption = max(
(r["retrieval_caption"] for r in chunk_results),
key=lambda x: x["score"],
)
return {
"tags": top_tags,
"retrieval_caption": best_caption,
}
def _encode_chunk(self, frames: np.ndarray, use_batching: bool = False) -> DeploymentResponse:
"""Encode a single chunk's frames to embeddings. Returns DeploymentResponse ref."""
return self.encoder.remote(frames, use_batching=use_batching)
async def _decode_chunk(
self,
encoder_output: dict,
chunk_index: int,
chunk_start_time: float,
chunk_duration: float,
ema_state=None,
) -> dict:
"""Decode embeddings to tags, caption, scene changes."""
return await self.decoder.remote(
encoder_output=encoder_output,
chunk_index=chunk_index,
chunk_start_time=chunk_start_time,
chunk_duration=chunk_duration,
ema_state=ema_state,
)
@fastapi_app.post("/analyze", response_model=AnalyzeResponse)
async def analyze(self, request: AnalyzeRequest) -> AnalyzeResponse:
"""
Analyze a video from S3 and return tags, caption, and scene changes.
Downloads video to temp file for fast local processing.
Chunks the entire video and aggregates results.
Encoder refs are passed directly to decoder for dependency resolution.
"""
total_start = time.perf_counter()
temp_path = None
try:
# Download video from S3 to temp file
download_start = time.perf_counter()
try:
temp_path = await self._download_video(request.video_path)
except Exception as e:
raise HTTPException(status_code=400, detail=f"Cannot download S3 video: {e}")
s3_download_ms = (time.perf_counter() - download_start) * 1000
# Chunk video with PARALLEL frame extraction from local file
decode_start = time.perf_counter()
try:
chunks = await chunk_video_async(
str(temp_path),
chunk_duration=request.chunk_duration,
num_frames_per_chunk=request.num_frames,
ffmpeg_threads=FFMPEG_THREADS,
use_single_ffmpeg=True,
)
except Exception as e:
raise HTTPException(status_code=400, detail=f"Cannot process video: {e}")
decode_video_ms = (time.perf_counter() - decode_start) * 1000
if not chunks:
raise HTTPException(status_code=400, detail="No chunks extracted from video")
# Calculate video duration from chunks
video_duration = chunks[-1].start_time + chunks[-1].duration
# Fire off all encoder calls (returns refs, not awaited)
encode_start = time.perf_counter()
encode_refs = [
self._encode_chunk(chunk.frames, use_batching=request.use_batching)
for chunk in chunks
]
encode_ms = (time.perf_counter() - encode_start) * 1000
# Decode chunks SERIALLY, passing encoder refs directly.
# Ray Serve resolves the encoder result when decoder needs it.
# EMA state is tracked here (not in decoder) to ensure continuity
# even when autoscaling routes requests to different replicas.
decode_start = time.perf_counter()
decode_results = []
ema_state = None # Will be initialized from first chunk's first frame
for chunk, enc_ref in zip(chunks, encode_refs):
dec_result = await self._decode_chunk(
encoder_output=enc_ref,
chunk_index=chunk.index,
chunk_start_time=chunk.start_time,
chunk_duration=chunk.duration,
ema_state=ema_state,
)
decode_results.append(dec_result)
ema_state = dec_result["ema_state"] # Carry forward for next chunk
decode_ms = (time.perf_counter() - decode_start) * 1000
# Collect results
chunk_results = []
per_chunk_results = []
all_scene_changes = []
for chunk, decoder_result in zip(chunks, decode_results):
chunk_results.append(decoder_result)
# Scene changes come directly from decoder
chunk_scene_changes = [
SceneChange(**sc) for sc in decoder_result["scene_changes"]
]
all_scene_changes.extend(chunk_scene_changes)
per_chunk_results.append(ChunkResult(
chunk_index=chunk.index,
start_time=chunk.start_time,
duration=chunk.duration,
tags=[TagResult(**t) for t in decoder_result["tags"]],
retrieval_caption=CaptionResult(**decoder_result["retrieval_caption"]),
scene_changes=chunk_scene_changes,
))
# Aggregate results
aggregated = self._aggregate_results(chunk_results)
total_ms = (time.perf_counter() - total_start) * 1000
return AnalyzeResponse(
stream_id=request.stream_id,
tags=[TagResult(**t) for t in aggregated["tags"]],
retrieval_caption=CaptionResult(**aggregated["retrieval_caption"]),
scene_changes=all_scene_changes,
num_scene_changes=len(all_scene_changes),
chunks=per_chunk_results,
num_chunks=len(chunks),
video_duration=video_duration,
timing_ms=TimingResult(
s3_download_ms=round(s3_download_ms, 2),
decode_video_ms=round(decode_video_ms, 2),
encode_ms=round(encode_ms, 2),
decode_ms=round(decode_ms, 2),
total_ms=round(total_ms, 2),
),
)
finally:
# Clean up temp file
if temp_path and temp_path.exists():
temp_path.unlink(missing_ok=True)
@fastapi_app.get("/health")
async def health(self):
"""Health check endpoint."""
return {"status": "healthy"}
encoder = VideoEncoder.bind()
decoder = MultiDecoder.bind(bucket=os.environ.get("S3_BUCKET"))
app = VideoAnalyzer.bind(encoder=encoder, decoder=decoder)
doc/source/serve/tutorials/video-analysis/assets/chunk_video_analysis.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 146 KiB

doc/source/serve/tutorials/video-analysis/assets/chunk_video_heatmap.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 96 KiB

doc/source/serve/tutorials/video-analysis/assets/decord_vs_ffmpeg_local.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 70 KiB

doc/source/serve/tutorials/video-analysis/assets/ffmpeg_threads_benchmark.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 57 KiB

doc/source/serve/tutorials/video-analysis/assets/gpu_utilization.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 121 KiB

doc/source/serve/tutorials/video-analysis/assets/latency_distribution_by_concurrency.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 114 KiB

doc/source/serve/tutorials/video-analysis/assets/latency_histograms.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 166 KiB

doc/source/serve/tutorials/video-analysis/assets/latency_over_time.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 710 KiB

doc/source/serve/tutorials/video-analysis/assets/num_replicas.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 121 KiB

doc/source/serve/tutorials/video-analysis/assets/ongoing.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 111 KiB

doc/source/serve/tutorials/video-analysis/assets/presigned_vs_direct_s3.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 74 KiB

doc/source/serve/tutorials/video-analysis/assets/processing_time_breakdown.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 64 KiB

doc/source/serve/tutorials/video-analysis/assets/processing_time_pie_charts.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 176 KiB

doc/source/serve/tutorials/video-analysis/assets/queue_len.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 109 KiB

doc/source/serve/tutorials/video-analysis/assets/single_vs_multi_ffmpeg.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 62 KiB

doc/source/serve/tutorials/video-analysis/assets/sync_vs_async_comparison.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 52 KiB

doc/source/serve/tutorials/video-analysis/assets/thread_pool_benchmark.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 63 KiB

doc/source/serve/tutorials/video-analysis/assets/throughput_analysis.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 97 KiB

doc/source/serve/tutorials/video-analysis/autoscaling_policy.py
+110
View File
@@ -0,0 +1,110 @@
"""
Application-level autoscaling policy for video processing pipeline.
Scaling strategy:
- VideoAnalyzer: scales based on its load (error_ratio = requests / capacity)
- VideoEncoder: scales based on its load, with floor = VideoAnalyzer replicas
- MultiDecoder: 0.5x VideoEncoder replicas
Error ratio formula:
error_ratio = total_ongoing_requests / (target_per_replica × current_replicas)
- error_ratio > 1.0 → over capacity → scale up
- error_ratio < 1.0 → under capacity → scale down
- error_ratio = 1.0 → at capacity → no change
"""
import math
from typing import Dict, Tuple
from ray.serve._private.common import DeploymentID
from ray.serve.config import AutoscalingContext
def _get_error_ratio(ctx: AutoscalingContext) -> float:
"""
Calculate error ratio: how much over/under target capacity we are.
Returns 1.0 when idle to maintain current replicas.
"""
target_per_replica = ctx.config.target_ongoing_requests or 1
total_requests = ctx.total_num_requests
current_replicas = ctx.current_num_replicas
if total_requests == 0:
return 1.0 # Idle: maintain current replicas
total_capacity = target_per_replica * current_replicas
return total_requests / total_capacity
def _scale_by_error_ratio(ctx: AutoscalingContext, floor: int = 0) -> int:
"""
Calculate target replicas based on error ratio.
Args:
ctx: Deployment autoscaling context
floor: Minimum replicas (in addition to capacity_adjusted_min)
Returns:
Target replica count, clamped to min/max limits
"""
error_ratio = _get_error_ratio(ctx)
# Scale current replicas by error ratio
target = int(math.ceil(ctx.current_num_replicas * error_ratio))
# Apply floor (e.g., encoder should have at least as many as analyzer)
target = max(target, floor)
# Clamp to configured limits
return max(
ctx.capacity_adjusted_min_replicas,
min(ctx.capacity_adjusted_max_replicas, target),
)
def _find_deployment(
contexts: Dict[DeploymentID, AutoscalingContext],
name: str,
) -> Tuple[DeploymentID, AutoscalingContext]:
"""Find deployment by name."""
for dep_id, ctx in contexts.items():
if dep_id.name == name:
return dep_id, ctx
raise KeyError(f"Deployment '{name}' not found")
def coordinated_scaling_policy(
contexts: Dict[DeploymentID, AutoscalingContext],
) -> Tuple[Dict[DeploymentID, int], Dict]:
"""
Coordinated scaling for video processing pipeline.
Scaling rules:
VideoAnalyzer: scale by its own load
VideoEncoder: scale by its own load, floor = analyzer replicas
MultiDecoder: 0.5x encoder replicas
"""
decisions: Dict[DeploymentID, int] = {}
# 1. VideoAnalyzer: scale by load
analyzer_id, analyzer_ctx = _find_deployment(contexts, "VideoAnalyzer")
analyzer_replicas = _scale_by_error_ratio(analyzer_ctx)
decisions[analyzer_id] = analyzer_replicas
# 2. VideoEncoder: scale by load, but at least as many as analyzer
encoder_id, encoder_ctx = _find_deployment(contexts, "VideoEncoder")
encoder_replicas = _scale_by_error_ratio(encoder_ctx, floor=analyzer_replicas)
decisions[encoder_id] = encoder_replicas
# 3. MultiDecoder: 0.5x encoder replicas
decoder_id, decoder_ctx = _find_deployment(contexts, "MultiDecoder")
decoder_replicas = max(1, math.ceil(encoder_replicas / 2))
decoder_replicas = max(
decoder_ctx.capacity_adjusted_min_replicas,
min(decoder_ctx.capacity_adjusted_max_replicas, decoder_replicas),
)
decisions[decoder_id] = decoder_replicas
return decisions, {}
doc/source/serve/tutorials/video-analysis/chunk_video_benchmark.ipynb
+1741
View File
File diff suppressed because one or more lines are too long
doc/source/serve/tutorials/video-analysis/ci/aws.yaml
+19
View File
@@ -0,0 +1,19 @@
cloud_id: {{ env["ANYSCALE_CLOUD_ID"] }}
region: us-west-2
# Cluster resources
head_node_type:
name: head
instance_type: m5.2xlarge
resources:
cpu: 8
worker_node_types:
- name: 16CPU-64GB
instance_type: m8i.4xlarge
min_workers: 1
max_workers: 32
- name: 4CPU-16GB-1GPU
instance_type: g6.xlarge
min_workers: 1
max_workers: 32
doc/source/serve/tutorials/video-analysis/ci/build.sh
Executable
+21
View File
@@ -0,0 +1,21 @@
#!/bin/bash
set -euxo pipefail
# Install Python dependencies
pip3 install --no-cache-dir \
"numpy>=1.24.0,<2.0" \
"torch>=2.0.0" \
"transformers>=4.35.0" \
"accelerate>=0.25.0" \
"sentencepiece>=0.1.99" \
"httpx>=0.25.0" \
"aioboto3>=12.0.0" \
"pillow>=10.0.0"
sudo apt-get update && sudo apt-get install -y --no-install-recommends \
ffmpeg \
&& sudo rm -rf /var/lib/apt/lists/*
export S3_BUCKET=anyscale-example-video-analysis-test-bucket
doc/source/serve/tutorials/video-analysis/ci/gce.yaml
+19
View File
@@ -0,0 +1,19 @@
cloud_id: {{ env["ANYSCALE_CLOUD_ID"] }} # required; set by CI
region: us-central1
# Cluster resources
head_node_type:
name: head
instance_type: n1-standard-8
resources:
cpu: 8
worker_node_types:
- name: 16CPU-64GB-GPU
instance_type: n4-standard-16
min_workers: 1
max_workers: 32
- name: 4CPU-16GB-GPU
instance_type: g2-standard-4
min_workers: 1
max_workers: 32
doc/source/serve/tutorials/video-analysis/ci/nb2py.py
+40
View File
@@ -0,0 +1,40 @@
#!/usr/bin/env python3
import argparse
import nbformat
def convert_notebook(input_path: str, output_path: str) -> None:
"""
Read a Jupyter notebook and write a Python script.
Cells that load or autoreload extensions are ignored.
"""
nb = nbformat.read(input_path, as_version=4)
with open(output_path, "w") as out:
for cell in nb.cells:
if cell.cell_type != "code":
continue
lines = cell.source.splitlines()
# Skip cells that load or autoreload extensions
if any(
l.strip().startswith("%load_ext autoreload")
or l.strip().startswith("%autoreload all")
for l in lines
):
continue
out.write(cell.source.rstrip() + "\n\n")
def main() -> None:
parser = argparse.ArgumentParser(
description="Convert a Jupyter notebook to a Python script."
)
parser.add_argument("input_nb", help="Path to the input .ipynb file")
parser.add_argument("output_py", help="Path for the output .py script")
args = parser.parse_args()
convert_notebook(args.input_nb, args.output_py)
if __name__ == "__main__":
main()
doc/source/serve/tutorials/video-analysis/ci/tests.sh
Executable
+11
View File
@@ -0,0 +1,11 @@
#!/bin/bash
set -euxo pipefail
S3_BUCKET=anyscale-example-video-analysis-test-bucket
export S3_BUCKET
python ci/nb2py.py README.ipynb README.py # convert notebook to py script
python README.py
rm README.py # remove the generated script
doc/source/serve/tutorials/video-analysis/client/__init__.py
View File
doc/source/serve/tutorials/video-analysis/client/load_test.py
+374
View File
@@ -0,0 +1,374 @@
#!/usr/bin/env python3
"""
Load testing script for the Ray Serve video API.
Usage:
python -m client.load_test --video s3://bucket/path/to/video.mp4 --concurrency 4
python -m client.load_test --video s3://bucket/video.mp4 --concurrency 8 --url http://localhost:8000
python -m client.load_test --video s3://bucket/video.mp4 --concurrency 4 --token YOUR_TOKEN
Press Ctrl+C to stop and save results to CSV.
"""
import argparse
import asyncio
import csv
import signal
import sys
import time
import uuid
from dataclasses import dataclass, field, asdict
from datetime import datetime
from pathlib import Path
from typing import Optional
import httpx
@dataclass
class RequestResult:
"""Result of a single request."""
request_id: str
stream_id: str
start_time: float
end_time: float
latency_ms: float
success: bool
status_code: int
error: Optional[str] = None
# Server-side timing (if successful)
s3_download_ms: Optional[float] = None
decode_video_ms: Optional[float] = None
encode_ms: Optional[float] = None
decode_ms: Optional[float] = None
server_total_ms: Optional[float] = None
# Response data
num_chunks: Optional[int] = None
video_duration: Optional[float] = None
num_scene_changes: Optional[int] = None
@dataclass
class LoadTestStats:
"""Aggregated statistics for the load test."""
total_requests: int = 0
successful_requests: int = 0
failed_requests: int = 0
total_latency_ms: float = 0.0
min_latency_ms: float = float('inf')
max_latency_ms: float = 0.0
latencies: list = field(default_factory=list)
def add_result(self, result: RequestResult):
self.total_requests += 1
if result.success:
self.successful_requests += 1
self.total_latency_ms += result.latency_ms
self.min_latency_ms = min(self.min_latency_ms, result.latency_ms)
self.max_latency_ms = max(self.max_latency_ms, result.latency_ms)
self.latencies.append(result.latency_ms)
else:
self.failed_requests += 1
@property
def avg_latency_ms(self) -> float:
if not self.latencies:
return 0.0
return self.total_latency_ms / len(self.latencies)
@property
def p50_latency_ms(self) -> float:
return self._percentile(50)
@property
def p95_latency_ms(self) -> float:
return self._percentile(95)
@property
def p99_latency_ms(self) -> float:
return self._percentile(99)
def _percentile(self, p: float) -> float:
if not self.latencies:
return 0.0
sorted_latencies = sorted(self.latencies)
idx = int(len(sorted_latencies) * p / 100)
idx = min(idx, len(sorted_latencies) - 1)
return sorted_latencies[idx]
class LoadTester:
def __init__(
self,
video_path: str,
url: str,
concurrency: int,
num_frames: int = 16,
chunk_duration: float = 10.0,
timeout: float = 300.0,
token: Optional[str] = None,
):
self.video_path = video_path
self.url = url
self.concurrency = concurrency
self.num_frames = num_frames
self.chunk_duration = chunk_duration
self.timeout = timeout
self.token = token
self.results: list[RequestResult] = []
self.stats = LoadTestStats()
self.running = True
self.start_time: Optional[float] = None
self.request_counter = 0
self._lock = asyncio.Lock()
self._semaphore: asyncio.Semaphore = None
def _build_payload(self) -> dict:
"""Build the request payload."""
stream_id = uuid.uuid4().hex[:8]
return {
"stream_id": stream_id,
"video_path": self.video_path,
"num_frames": self.num_frames,
"chunk_duration": self.chunk_duration,
"use_batching": False,
}
async def _make_request(self, client: httpx.AsyncClient, request_id: str) -> RequestResult:
"""Make a single request and return the result."""
payload = self._build_payload()
stream_id = payload["stream_id"]
start = time.perf_counter()
start_timestamp = time.time()
headers = {}
if self.token:
headers["Authorization"] = f"Bearer {self.token}"
try:
response = await client.post(f"{self.url}/analyze", json=payload, headers=headers)
end = time.perf_counter()
latency_ms = (end - start) * 1000
if response.status_code == 200:
data = response.json()
timing = data.get("timing_ms", {})
return RequestResult(
request_id=request_id,
stream_id=stream_id,
start_time=start_timestamp,
end_time=time.time(),
latency_ms=latency_ms,
success=True,
status_code=response.status_code,
s3_download_ms=timing.get("s3_download_ms"),
decode_video_ms=timing.get("decode_video_ms"),
encode_ms=timing.get("encode_ms"),
decode_ms=timing.get("decode_ms"),
server_total_ms=timing.get("total_ms"),
num_chunks=data.get("num_chunks"),
video_duration=data.get("video_duration"),
num_scene_changes=data.get("num_scene_changes"),
)
else:
return RequestResult(
request_id=request_id,
stream_id=stream_id,
start_time=start_timestamp,
end_time=time.time(),
latency_ms=latency_ms,
success=False,
status_code=response.status_code,
error=response.text[:200],
)
except Exception as e:
end = time.perf_counter()
latency_ms = (end - start) * 1000
return RequestResult(
request_id=request_id,
stream_id=stream_id,
start_time=start_timestamp,
end_time=time.time(),
latency_ms=latency_ms,
success=False,
status_code=0,
error=str(e)[:200],
)
async def _request_task(self, client: httpx.AsyncClient, request_id: str):
"""Execute a single request and record the result. Releases semaphore when done."""
try:
result = await self._make_request(client, request_id)
async with self._lock:
self.results.append(result)
self.stats.add_result(result)
# Print progress
status = "✓" if result.success else "✗"
print(
f"{status} {result.request_id} | "
f"{result.latency_ms:7.1f}ms | "
f"Total: {self.stats.total_requests} "
f"(OK: {self.stats.successful_requests}, Fail: {self.stats.failed_requests})"
)
finally:
self._semaphore.release()
async def run(self):
"""Run the load test until interrupted."""
self.start_time = time.time()
self._semaphore = asyncio.Semaphore(self.concurrency)
pending_tasks: set[asyncio.Task] = set()
print("=" * 70)
print("🚀 Starting Load Test")
print("=" * 70)
print(f" Video: {self.video_path}")
print(f" URL: {self.url}")
print(f" Concurrency: {self.concurrency}")
print(f" Chunk dur: {self.chunk_duration}s")
print(f" Frames/chunk:{self.num_frames}")
print()
print("Press Ctrl+C to stop and save results...")
print("-" * 70)
async with httpx.AsyncClient(timeout=self.timeout) as client:
while self.running:
# Block until a slot is available
await self._semaphore.acquire()
if not self.running:
self._semaphore.release()
break
# Spawn task (it will release semaphore when done)
self.request_counter += 1
request_id = f"req_{self.request_counter:06d}"
task = asyncio.create_task(self._request_task(client, request_id))
pending_tasks.add(task)
task.add_done_callback(pending_tasks.discard)
# Wait for in-flight requests
if pending_tasks:
print(f"\n⏳ Waiting for {len(pending_tasks)} in-flight requests...")
await asyncio.gather(*pending_tasks, return_exceptions=True)
def stop(self):
"""Stop the load test."""
self.running = False
def save_results(self, output_path: str):
"""Save results to CSV."""
if not self.results:
print("No results to save.")
return
fieldnames = [
"request_id", "stream_id", "start_time", "end_time", "latency_ms",
"success", "status_code", "error",
"s3_download_ms", "decode_video_ms", "encode_ms", "decode_ms", "server_total_ms",
"num_chunks", "video_duration", "num_scene_changes"
]
with open(output_path, "w", newline="") as f:
writer = csv.DictWriter(f, fieldnames=fieldnames)
writer.writeheader()
for result in self.results:
writer.writerow(asdict(result))
print(f"📁 Results saved to: {output_path}")
def print_summary(self):
"""Print summary statistics."""
if not self.results:
print("No results to summarize.")
return
duration = time.time() - self.start_time if self.start_time else 0
throughput = self.stats.total_requests / duration if duration > 0 else 0
print()
print("=" * 70)
print("📊 Load Test Summary")
print("=" * 70)
print(f" Duration: {duration:.1f}s")
print(f" Total requests: {self.stats.total_requests}")
print(f" Successful: {self.stats.successful_requests}")
print(f" Failed: {self.stats.failed_requests}")
print(f" Success rate: {self.stats.successful_requests / self.stats.total_requests * 100:.1f}%")
print(f" Throughput: {throughput:.2f} req/s")
print()
print("⏱️ Latency (successful requests):")
if self.stats.latencies:
print(f" Min: {self.stats.min_latency_ms:8.1f} ms")
print(f" Max: {self.stats.max_latency_ms:8.1f} ms")
print(f" Avg: {self.stats.avg_latency_ms:8.1f} ms")
print(f" P50: {self.stats.p50_latency_ms:8.1f} ms")
print(f" P95: {self.stats.p95_latency_ms:8.1f} ms")
print(f" P99: {self.stats.p99_latency_ms:8.1f} ms")
else:
print(" (no successful requests)")
print("=" * 70)
def main():
parser = argparse.ArgumentParser(description="Load test the Ray Serve video API")
parser.add_argument("--video", type=str, required=True, help="S3 URI: s3://bucket/key")
parser.add_argument("--url", type=str, default="http://127.0.0.1:8000", help="Server URL")
parser.add_argument("--concurrency", type=int, default=4, help="Number of concurrent workers")
parser.add_argument("--num-frames", type=int, default=16, help="Frames per chunk")
parser.add_argument("--chunk-duration", type=float, default=10.0, help="Chunk duration in seconds")
parser.add_argument("--timeout", type=float, default=300.0, help="Request timeout in seconds")
parser.add_argument("--output", type=str, default=None, help="Output CSV path (default: load_test_<timestamp>.csv)")
parser.add_argument("--token", type=str, default=None, help="Bearer token for Authorization header")
args = parser.parse_args()
# Generate output path if not provided
if args.output is None:
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
args.output = f"load_test_{timestamp}_{args.concurrency}.csv"
tester = LoadTester(
video_path=args.video,
url=args.url,
concurrency=args.concurrency,
num_frames=args.num_frames,
chunk_duration=args.chunk_duration,
timeout=args.timeout,
token=args.token,
)
# Track interrupt count for force exit
interrupt_count = 0
# Handle Ctrl+C gracefully
def signal_handler(sig, frame):
nonlocal interrupt_count
interrupt_count += 1
if interrupt_count == 1:
print("\n\n🛑 Stopping load test (press Ctrl+C again to force exit)...")
tester.stop()
else:
print("\n\n⚠️ Force exiting...")
tester.print_summary()
tester.save_results(args.output)
sys.exit(1)
signal.signal(signal.SIGINT, signal_handler)
signal.signal(signal.SIGTERM, signal_handler)
try:
asyncio.run(tester.run())
except KeyboardInterrupt:
pass
finally:
tester.print_summary()
tester.save_results(args.output)
if __name__ == "__main__":
main()
doc/source/serve/tutorials/video-analysis/client/send_video.py
+115
View File
@@ -0,0 +1,115 @@
#!/usr/bin/env python3
"""
Client script to send video to the Ray Serve API.
Usage:
python -m client.send_video --video s3://bucket/path/to/video.mp4
python -m client.send_video --video s3://bucket/video.mp4 --chunk-duration 5.0
python -m client.send_video --video s3://bucket/video.mp4 --token YOUR_TOKEN
"""
import argparse
import time
import uuid
import httpx
def main():
parser = argparse.ArgumentParser(description="Send video to Ray Serve API")
parser.add_argument("--video", type=str, required=True, help="S3 URI: s3://bucket/key")
parser.add_argument("--stream-id", type=str, default=None, help="Stream ID (random if not provided)")
parser.add_argument("--num-frames", type=int, default=16, help="Frames per chunk")
parser.add_argument("--chunk-duration", type=float, default=10.0, help="Chunk duration in seconds")
parser.add_argument("--url", type=str, default="http://127.0.0.1:8000", help="Server URL")
parser.add_argument("--token", type=str, default=None, help="Bearer token for Authorization header")
args = parser.parse_args()
# Generate random stream ID if not provided
stream_id = args.stream_id or uuid.uuid4().hex[:8]
payload = {
"stream_id": stream_id,
"video_path": args.video,
"num_frames": args.num_frames,
"chunk_duration": args.chunk_duration,
"use_batching": True
}
print(f"📹 Processing video: {args.video}")
print(f" Stream ID: {stream_id}")
print(f" Chunk duration: {args.chunk_duration}s, Frames/chunk: {args.num_frames}")
print()
start = time.perf_counter()
headers = {}
if args.token:
headers["Authorization"] = f"Bearer {args.token}"
with httpx.Client(timeout=300.0) as client:
response = client.post(f"{args.url}/analyze", json=payload, headers=headers)
latency_ms = (time.perf_counter() - start) * 1000
if response.status_code != 200:
print(f"❌ Error {response.status_code}: {response.text}")
return
result = response.json()
print("=" * 60)
print("✅ Response")
print("=" * 60)
print(f"Stream ID: {result['stream_id']}")
print(f"Video duration: {result['video_duration']:.1f}s")
print(f"Chunks processed: {result['num_chunks']}")
print()
print("🏷️ Top Tags (aggregated):")
for tag in result["tags"]:
print(f" {tag['score']:.3f} {tag['text']}")
print()
print("📝 Best Caption:")
caption = result["retrieval_caption"]
print(f" {caption['score']:.3f} {caption['text']}")
print()
# Scene changes
scene_changes = result["scene_changes"]
print(f"🎬 Scene Changes Detected: {result['num_scene_changes']}")
if scene_changes:
for sc in scene_changes:
print(f" {sc['timestamp']:6.2f}s score={sc['score']:.3f} (chunk {sc['chunk_index']}, frame {sc['frame_index']})")
else:
print(" (none detected)")
print()
# Show per-chunk results
print("📊 Per-Chunk Results:")
print("-" * 60)
for chunk in result["chunks"]:
print(f" Chunk {chunk['chunk_index']}: {chunk['start_time']:.1f}s - {chunk['start_time'] + chunk['duration']:.1f}s")
print(f" Top tag: {chunk['tags'][0]['text']} ({chunk['tags'][0]['score']:.3f})")
print(f" Caption: {chunk['retrieval_caption']['text'][:50]}...")
num_changes = len(chunk["scene_changes"])
print(f" Scene changes: {num_changes}")
print()
timing = result["timing_ms"]
print("⏱️ Timing:")
print(f" S3 download: {timing['s3_download_ms']:.1f} ms")
print(f" Video decode: {timing['decode_video_ms']:.1f} ms")
print(f" Encode (GPU): {timing['encode_ms']:.1f} ms")
print(f" Decode (CPU): {timing['decode_ms']:.1f} ms")
print(f" Total server: {timing['total_ms']:.1f} ms")
print(f" Round-trip: {latency_ms:.1f} ms")
if result['num_chunks'] > 1:
avg_per_chunk = timing['total_ms'] / result['num_chunks']
print(f" Avg/chunk: {avg_per_chunk:.1f} ms")
if __name__ == "__main__":
main()
doc/source/serve/tutorials/video-analysis/config.yaml
+91
View File
@@ -0,0 +1,91 @@
# This file was generated using the `serve build` command on Ray v2.53.0.
proxy_location: EveryNode
http_options:
host: 0.0.0.0
port: 8000
grpc_options:
port: 9000
grpc_servicer_functions: []
logging_config:
encoding: JSON
log_level: INFO
logs_dir: null
enable_access_log: true
additional_log_standard_attrs: []
applications:
- name: app1
route_prefix: /
import_path: app:app
runtime_env: {}
autoscaling_policy:
policy_function: autoscaling_policy:coordinated_scaling_policy
deployments:
- name: VideoEncoder
num_replicas: "auto"
max_ongoing_requests: 2
autoscaling_config:
min_replicas: 1
initial_replicas: null
max_replicas: 32
target_ongoing_requests: 2.0
metrics_interval_s: 10.0
look_back_period_s: 30.0
smoothing_factor: 1.0
upscale_smoothing_factor: null
downscale_smoothing_factor: null
upscaling_factor: null
downscaling_factor: null
downscale_delay_s: 600.0
downscale_to_zero_delay_s: null
upscale_delay_s: 30.0
aggregation_function: mean
ray_actor_options:
num_cpus: 2.0
num_gpus: 1.0
- name: MultiDecoder
num_replicas: "auto"
max_ongoing_requests: 4
autoscaling_config:
min_replicas: 1
initial_replicas: null
max_replicas: 32
target_ongoing_requests: 2.0
metrics_interval_s: 10.0
look_back_period_s: 30.0
smoothing_factor: 1.0
upscale_smoothing_factor: null
downscale_smoothing_factor: null
upscaling_factor: null
downscaling_factor: null
downscale_delay_s: 600.0
downscale_to_zero_delay_s: null
upscale_delay_s: 30.0
aggregation_function: mean
ray_actor_options:
num_cpus: 1.0
- name: VideoAnalyzer
num_replicas: "auto"
max_ongoing_requests: 4
autoscaling_config:
min_replicas: 1
initial_replicas: null
max_replicas: 64
target_ongoing_requests: 2.0
metrics_interval_s: 10.0
look_back_period_s: 30.0
smoothing_factor: 1.0
upscale_smoothing_factor: null
downscale_smoothing_factor: null
upscaling_factor: null
downscaling_factor: null
downscale_delay_s: 600.0
downscale_to_zero_delay_s: null
upscale_delay_s: 30.0
aggregation_function: mean
ray_actor_options:
num_cpus: 6.0
doc/source/serve/tutorials/video-analysis/constants.py
+60
View File
@@ -0,0 +1,60 @@
# Project constants
# SigLIP model
MODEL_NAME = "google/siglip-so400m-patch14-384"
# S3 paths
S3_VIDEOS_PREFIX = "stock-videos/"
S3_EMBEDDINGS_PREFIX = "embeddings/"
# Scene change detection
SCENE_CHANGE_THRESHOLD = 0.15 # EMA score threshold for detecting scene changes
EMA_ALPHA = 0.9 # EMA decay factor (higher = slower adaptation)
# Pexels API
PEXELS_API_BASE = "https://api.pexels.com/videos"
# Concurrency limits
MAX_CONCURRENT_DOWNLOADS = 5
MAX_CONCURRENT_UPLOADS = 5
# Video normalization defaults (384x384 matches model input size for fastest inference)
NORMALIZE_WIDTH = 384
NORMALIZE_HEIGHT = 384
NORMALIZE_FPS = 30
# Video search queries (for downloading stock videos)
SEARCH_QUERIES = [
"kitchen cooking",
"office meeting",
"street city traffic",
"living room home",
"restaurant cafe",
"parking lot cars",
"classroom students",
"warehouse industrial",
"grocery store shopping",
"gym exercise workout",
"person speaking",
"crowd people walking",
"laptop computer work",
"outdoor nature",
"presentation business",
"conversation talking",
"running jogging",
"dining food",
"shopping mall",
"park outdoor",
]
# Video chunking defaults
DEFAULT_NUM_FRAMES = 16
DEFAULT_CHUNK_DURATION = 10.0
# FFmpeg configuration, restricting to 2 threads to avoid over subscription.
FFMPEG_THREADS = 6
# Setting this to 2 because i am assuming average
# video length in my corpus is 20 seconds.
# 20/10(default chunk duration) = 2
# this means we want to set num_cpus to 4 for deployment.
NUM_WORKERS = 3
doc/source/serve/tutorials/video-analysis/deployments/__init__.py
View File
doc/source/serve/tutorials/video-analysis/deployments/decoder.py
+205
View File
@@ -0,0 +1,205 @@
"""MultiDecoder deployment - CPU-based classification, retrieval, and scene detection."""
import io
import logging
import os
import aioboto3
import numpy as np
from ray import serve
from constants import (
S3_EMBEDDINGS_PREFIX,
SCENE_CHANGE_THRESHOLD,
EMA_ALPHA,
)
from utils.s3 import get_s3_region
logger = logging.getLogger(__name__)
@serve.deployment(
num_replicas="auto",
ray_actor_options={"num_cpus": 1},
max_ongoing_requests=4, # can be set higher than 4, but since the encoder is limited to 4, we need to keep it at 4.
autoscaling_config={
"min_replicas": 1,
"max_replicas": 10,
"target_num_ongoing_requests": 2,
},
)
class MultiDecoder:
"""
Decodes video embeddings into tags, captions, and scene changes.
Uses precomputed text embeddings loaded from S3.
This deployment is stateless - EMA state for scene detection is passed
in and returned with each call, allowing the caller to maintain state
continuity across multiple replicas.
"""
async def __init__(self, bucket: str, s3_prefix: str = S3_EMBEDDINGS_PREFIX):
"""Initialize decoder with text embeddings from S3."""
self.bucket = bucket
self.ema_alpha = EMA_ALPHA
self.scene_threshold = SCENE_CHANGE_THRESHOLD
self.s3_prefix = s3_prefix
logger.info(f"MultiDecoder initializing (bucket={self.bucket}, ema_alpha={self.ema_alpha}, threshold={self.scene_threshold})")
await self._load_embeddings()
logger.info(f"MultiDecoder ready (tags={len(self.tag_texts)}, descriptions={len(self.desc_texts)})")
async def _load_embeddings(self):
"""Load precomputed text embeddings from S3."""
session = aioboto3.Session(region_name=get_s3_region(self.bucket))
async with session.client("s3") as s3:
# Load tag embeddings
tag_key = f"{self.s3_prefix}tag_embeddings.npz"
response = await s3.get_object(Bucket=self.bucket, Key=tag_key)
tag_data = await response["Body"].read()
tag_npz = np.load(io.BytesIO(tag_data), allow_pickle=True)
self.tag_embeddings = tag_npz["embeddings"]
self.tag_texts = tag_npz["texts"].tolist()
# Load description embeddings
desc_key = f"{self.s3_prefix}description_embeddings.npz"
response = await s3.get_object(Bucket=self.bucket, Key=desc_key)
desc_data = await response["Body"].read()
desc_npz = np.load(io.BytesIO(desc_data), allow_pickle=True)
self.desc_embeddings = desc_npz["embeddings"]
self.desc_texts = desc_npz["texts"].tolist()
def _cosine_similarity(self, embedding: np.ndarray, bank: np.ndarray) -> np.ndarray:
"""Compute cosine similarity between embedding and all vectors in bank."""
return bank @ embedding
def _get_top_tags(self, embedding: np.ndarray, top_k: int = 5) -> list[dict]:
"""Get top-k matching tags with scores."""
scores = self._cosine_similarity(embedding, self.tag_embeddings)
top_indices = np.argsort(scores)[::-1][:top_k]
return [
{"text": self.tag_texts[i], "score": float(scores[i])}
for i in top_indices
]
def _get_retrieval_caption(self, embedding: np.ndarray) -> dict:
"""Get best matching description."""
scores = self._cosine_similarity(embedding, self.desc_embeddings)
best_idx = np.argmax(scores)
return {
"text": self.desc_texts[best_idx],
"score": float(scores[best_idx]),
}
def _detect_scene_changes(
self,
frame_embeddings: np.ndarray,
chunk_index: int,
chunk_start_time: float,
chunk_duration: float,
ema_state: np.ndarray | None = None,
) -> tuple[list[dict], np.ndarray]:
"""
Detect scene changes using EMA-based scoring.
score_t = 1 - cosine(E_t, ema_t)
ema_t = α * ema_{t-1} + (1-α) * E_t
Args:
frame_embeddings: (T, D) normalized embeddings
chunk_index: Index of this chunk in the video
chunk_start_time: Start time of chunk in video (seconds)
chunk_duration: Duration of chunk (seconds)
ema_state: EMA state from previous chunk, or None for first chunk
Returns:
Tuple of (scene_changes list, updated ema_state)
"""
num_frames = len(frame_embeddings)
if num_frames == 0:
# Return empty changes and unchanged state (or zeros if no state)
return [], ema_state if ema_state is not None else np.zeros(0)
# Initialize EMA from first frame if no prior state
ema = ema_state.copy() if ema_state is not None else frame_embeddings[0].copy()
scene_changes = []
for frame_idx, embedding in enumerate(frame_embeddings):
# Compute score: how different is current frame from recent history
similarity = float(np.dot(embedding, ema))
score = max(0.0, 1.0 - similarity)
# Detect scene change if score exceeds threshold
if score >= self.scene_threshold:
# Calculate timestamp within video
frame_offset = (frame_idx / max(1, num_frames - 1)) * chunk_duration
timestamp = chunk_start_time + frame_offset
scene_changes.append({
"timestamp": round(timestamp, 3),
"score": round(score, 4),
"chunk_index": chunk_index,
"frame_index": frame_idx,
})
# Update EMA
ema = self.ema_alpha * ema + (1 - self.ema_alpha) * embedding
# Re-normalize
ema = ema / np.linalg.norm(ema)
return scene_changes, ema
def __call__(
self,
encoder_output: dict,
chunk_index: int,
chunk_start_time: float,
chunk_duration: float,
top_k_tags: int = 5,
ema_state: np.ndarray | None = None,
) -> dict:
"""
Decode embeddings into tags, caption, and scene changes.
Args:
encoder_output: Dict with 'frame_embeddings' and 'embedding_dim'
chunk_index: Index of this chunk in the video
chunk_start_time: Start time of chunk (seconds)
chunk_duration: Duration of chunk (seconds)
top_k_tags: Number of top tags to return
ema_state: EMA state from previous chunk for scene detection continuity.
Pass None for the first chunk of a stream.
Returns:
Dict containing tags, retrieval_caption, scene_changes, and updated ema_state.
The caller should pass the returned ema_state to the next chunk's call.
"""
# Get frame embeddings from encoder output
frame_embeddings = encoder_output["frame_embeddings"]
# Calculate pooled embedding (mean across frames, normalized)
pooled_embedding = frame_embeddings.mean(axis=0)
pooled_embedding = pooled_embedding / np.linalg.norm(pooled_embedding)
# Classification and retrieval on pooled embedding
tags = self._get_top_tags(pooled_embedding, top_k=top_k_tags)
caption = self._get_retrieval_caption(pooled_embedding)
# Scene change detection on frame embeddings
scene_changes, new_ema_state = self._detect_scene_changes(
frame_embeddings=frame_embeddings,
chunk_index=chunk_index,
chunk_start_time=chunk_start_time,
chunk_duration=chunk_duration,
ema_state=ema_state,
)
return {
"tags": tags,
"retrieval_caption": caption,
"scene_changes": scene_changes,
"ema_state": new_ema_state,
}
doc/source/serve/tutorials/video-analysis/deployments/encoder.py
+140
View File
@@ -0,0 +1,140 @@
"""VideoEncoder deployment - GPU-based frame encoding using SigLIP."""
import asyncio
import logging
from typing import List
import numpy as np
import torch
from ray import serve
from transformers import AutoModel, AutoProcessor
from constants import MODEL_NAME
from utils.video import frames_to_pil_list
logger = logging.getLogger(__name__)
@serve.deployment(
num_replicas="auto",
ray_actor_options={"num_gpus": 1, "num_cpus": 2},
# GPU utilization is at 100% when this is set to 2. with L4
# aka number on ongoing chunks that can be processed at once.
max_ongoing_requests=2,
autoscaling_config={
"min_replicas": 1,
"max_replicas": 10,
"target_num_ongoing_requests": 2,
},
)
class VideoEncoder:
"""
Encodes video frames into embeddings using SigLIP.
Returns both per-frame embeddings and pooled embedding.
"""
def __init__(self):
self.device = "cuda" if torch.cuda.is_available() else "cpu"
print(f"VideoEncoder initializing on {self.device}")
# Load SigLIP model and processor
self.processor = AutoProcessor.from_pretrained(MODEL_NAME)
self.model = AutoModel.from_pretrained(MODEL_NAME).to(self.device)
self.model.eval()
# Get embedding dimension
self.embedding_dim = self.model.config.vision_config.hidden_size
print(f"VideoEncoder ready (embedding_dim={self.embedding_dim})")
def encode_frames(self, frames: np.ndarray) -> np.ndarray:
"""
Encode frames and return per-frame embeddings.
Args:
frames: np.ndarray of shape (T, H, W, 3) uint8 RGB
Returns:
np.ndarray of shape (T, D) float32, L2-normalized per-frame embeddings
"""
# Convert to PIL images
pil_images = frames_to_pil_list(frames)
# Process images
inputs = self.processor(images=pil_images, return_tensors="pt").to(self.device)
# Get embeddings
with torch.no_grad():
with torch.amp.autocast(device_type=self.device, enabled=self.device == "cuda"):
outputs = self.model.get_image_features(**inputs)
# get_image_features returns BaseModelOutputWithPooling; use pooler_output for embeddings
frame_embeddings = torch.nn.functional.normalize(outputs.pooler_output, p=2, dim=1)
# Move to CPU and convert to numpy
result = frame_embeddings.cpu().numpy().astype(np.float32)
return result
async def encode_unbatched(self, frames: np.ndarray) -> dict:
"""
Unbatched entry point - processes single request directly.
Args:
frames: np.ndarray of shape (T, H, W, 3)
Returns:
dict with 'frame_embeddings' and 'embedding_dim'
"""
print(f"Unbatched: {frames.shape[0]} frames")
frame_embeddings = await asyncio.to_thread(self.encode_frames, frames)
return {
"frame_embeddings": frame_embeddings,
"embedding_dim": self.embedding_dim,
}
@serve.batch(max_batch_size=2, batch_wait_timeout_s=0.1)
async def encode_batched(self, frames_batch: List[np.ndarray]) -> List[dict]:
"""
Batched entry point - collects multiple requests into single GPU call.
Args:
frames_batch: List of frame arrays, each of shape (T, H, W, 3)
Returns:
List of dicts, each with 'frame_embeddings' and 'embedding_dim'
"""
frame_counts = [f.shape[0] for f in frames_batch]
total_frames = sum(frame_counts)
print(f"Batched: {len(frames_batch)} requests ({total_frames} total frames)")
# Concatenate all frames into single batch
all_frames = np.concatenate(frames_batch, axis=0)
# Single forward pass for all frames
all_embeddings = await asyncio.to_thread(self.encode_frames, all_frames)
# Split results back per request
results = []
offset = 0
for n_frames in frame_counts:
chunk_embeddings = all_embeddings[offset:offset + n_frames]
results.append({
"frame_embeddings": chunk_embeddings,
"embedding_dim": self.embedding_dim,
})
offset += n_frames
return results
async def __call__(self, frames: np.ndarray, use_batching: bool = False) -> dict:
"""
Main entry point. Set use_batching=False for direct comparison.
"""
if use_batching:
return await self.encode_batched(frames)
else:
return await self.encode_unbatched(frames)
doc/source/serve/tutorials/video-analysis/jobs/__init__.py
View File
doc/source/serve/tutorials/video-analysis/jobs/generate_text_embeddings.py
+195
View File
@@ -0,0 +1,195 @@
#!/usr/bin/env python3
"""
Ray job to generate SigLIP text embeddings for tags and descriptions.
Usage:
ray job submit --working-dir . -- python jobs/generate_text_embeddings.py --bucket my-bucket
"""
import argparse
import asyncio
import io
import json
import time
import aioboto3
import numpy as np
import ray
import torch
from transformers import AutoModel, AutoProcessor
from constants import MODEL_NAME, S3_EMBEDDINGS_PREFIX
from textbanks import TAGS, DESCRIPTIONS
from utils.s3 import get_s3_region
def load_textbanks() -> tuple[list[str], list[str]]:
"""Load tags and descriptions from textbanks module."""
return TAGS, DESCRIPTIONS
def compute_text_embeddings(
texts: list[str],
processor,
model,
device: str,
batch_size: int = 32,
) -> np.ndarray:
"""Compute normalized text embeddings using SigLIP."""
all_embeddings = []
for i in range(0, len(texts), batch_size):
batch_texts = texts[i:i + batch_size]
# Process text
inputs = processor(
text=batch_texts,
padding="max_length",
truncation=True,
return_tensors="pt",
)
inputs = {k: v.to(device) for k, v in inputs.items()}
# Get embeddings
with torch.no_grad():
outputs = model.get_text_features(**inputs)
# Handle case where outputs is a model output object vs raw tensor
if hasattr(outputs, 'pooler_output'):
embeddings = outputs.pooler_output.cpu().numpy()
else:
embeddings = outputs.cpu().numpy()
embeddings = embeddings / np.linalg.norm(embeddings, axis=1, keepdims=True)
all_embeddings.append(embeddings)
return np.vstack(all_embeddings).astype(np.float32)
async def save_to_s3(
session: aioboto3.Session,
embeddings: np.ndarray,
texts: list[str],
bucket: str,
key: str,
) -> str:
"""Save embeddings and texts to S3 as npz file."""
# Save as npz (embeddings + texts)
buffer = io.BytesIO()
np.savez_compressed(
buffer,
embeddings=embeddings,
texts=np.array(texts, dtype=object),
)
buffer.seek(0)
async with session.client("s3") as s3:
await s3.put_object(
Bucket=bucket,
Key=key,
Body=buffer.getvalue(),
ContentType="application/octet-stream",
)
return f"s3://{bucket}/{key}"
async def generate_and_upload(
bucket: str,
s3_prefix: str = S3_EMBEDDINGS_PREFIX,
) -> dict:
"""Generate embeddings and upload to S3."""
print("=" * 60)
print("Starting text embedding generation")
print("=" * 60)
# Load textbanks
print("\n📚 Loading text banks...")
tags, descriptions = load_textbanks()
print(f" Tags: {len(tags)}")
print(f" Descriptions: {len(descriptions)}")
# Load model
print(f"\n🤖 Loading SigLIP model: {MODEL_NAME}")
device = "cuda" if torch.cuda.is_available() else "cpu"
print(f" Device: {device}")
start = time.time()
processor = AutoProcessor.from_pretrained(MODEL_NAME)
model = AutoModel.from_pretrained(MODEL_NAME).to(device)
model.eval()
load_time = time.time() - start
print(f" Model loaded in {load_time:.1f}s")
# Generate tag embeddings
print("\n🏷️ Generating tag embeddings...")
start = time.time()
tag_embeddings = compute_text_embeddings(tags, processor, model, device)
tag_time = time.time() - start
print(f" Shape: {tag_embeddings.shape}")
print(f" Time: {tag_time:.2f}s")
# Generate description embeddings
print("\n📝 Generating description embeddings...")
start = time.time()
desc_embeddings = compute_text_embeddings(descriptions, processor, model, device)
desc_time = time.time() - start
print(f" Shape: {desc_embeddings.shape}")
print(f" Time: {desc_time:.2f}s")
# Upload to S3 concurrently
print(f"\n☁️ Uploading to S3 bucket: {bucket}")
session = aioboto3.Session(region_name=get_s3_region(bucket))
tag_uri, desc_uri = await asyncio.gather(
save_to_s3(session, tag_embeddings, tags, bucket, f"{s3_prefix}tag_embeddings.npz"),
save_to_s3(session, desc_embeddings, descriptions, bucket, f"{s3_prefix}description_embeddings.npz"),
)
print(f" Tags: {tag_uri}")
print(f" Descriptions: {desc_uri}")
print("\n✅ Done!")
return {
"tag_embeddings": {
"s3_uri": tag_uri,
"shape": list(tag_embeddings.shape),
"count": len(tags),
},
"description_embeddings": {
"s3_uri": desc_uri,
"shape": list(desc_embeddings.shape),
"count": len(descriptions),
},
"model": MODEL_NAME,
"timing": {
"model_load_s": load_time,
"tag_embed_s": tag_time,
"desc_embed_s": desc_time,
},
}
@ray.remote(num_gpus=1)
def generate_embeddings_task(bucket: str, s3_prefix: str = S3_EMBEDDINGS_PREFIX) -> dict:
"""Ray task wrapper for async embedding generation."""
return asyncio.run(generate_and_upload(bucket, s3_prefix))
def main():
parser = argparse.ArgumentParser(description="Generate text embeddings for decoder")
parser.add_argument("--bucket", type=str, required=True, help="S3 bucket name")
args = parser.parse_args()
# Run the task on existing Ray cluster
result = ray.get(generate_embeddings_task.remote(args.bucket))
print("\n" + "=" * 60)
print("Results:")
print("=" * 60)
print(json.dumps(result, indent=2))
if __name__ == "__main__":
main()
doc/source/serve/tutorials/video-analysis/scripts/__init__.py
View File
doc/source/serve/tutorials/video-analysis/scripts/download_stock_videos.py
+635
View File
@@ -0,0 +1,635 @@
#!/usr/bin/env python3
"""
Download stock videos from Pexels and upload to S3 (async version).
Videos are normalized to consistent specs before upload for predictable performance.
Usage:
python scripts/download_stock_videos.py --api-key YOUR_KEY --bucket YOUR_BUCKET
"""
import argparse
import asyncio
import json
import os
import shutil
import subprocess
import sys
import tempfile
from pathlib import Path
from typing import Optional
import aioboto3
import httpx
from botocore.exceptions import ClientError
from constants import (
SEARCH_QUERIES,
PEXELS_API_BASE,
MAX_CONCURRENT_DOWNLOADS,
MAX_CONCURRENT_UPLOADS,
S3_VIDEOS_PREFIX,
NORMALIZE_WIDTH,
NORMALIZE_HEIGHT,
NORMALIZE_FPS,
)
from utils.s3 import get_s3_region
def normalize_video(
input_path: Path,
output_path: Path,
width: int = NORMALIZE_WIDTH,
height: int = NORMALIZE_HEIGHT,
fps: int = NORMALIZE_FPS,
preset: str = "fast",
) -> bool:
"""
Normalize video to consistent specs using ffmpeg.
Applies:
- Resolution scaling with letterboxing to preserve aspect ratio
- Consistent FPS
- H.264 codec with main profile
- 1-second GOP for fast seeking
- Removes audio
Args:
input_path: Path to input video
output_path: Path for normalized output
width: Target width (default 1280)
height: Target height (default 720)
fps: Target FPS (default 30)
preset: x264 encoding preset (default "fast")
Returns:
True if successful, False otherwise
"""
cmd = [
"ffmpeg", "-y",
"-i", str(input_path),
"-vf", f"scale={width}:{height}:force_original_aspect_ratio=decrease,pad={width}:{height}:(ow-iw)/2:(oh-ih)/2,fps={fps}",
"-c:v", "libx264",
"-pix_fmt", "yuv420p",
"-profile:v", "main",
"-preset", preset,
"-g", str(fps), # GOP size = 1 second
"-keyint_min", str(fps),
"-sc_threshold", "0",
"-movflags", "+faststart",
"-an", # Remove audio
str(output_path),
]
result = subprocess.run(cmd, capture_output=True, text=True)
if result.returncode != 0:
print(f" ⚠️ ffmpeg normalization failed: {result.stderr[:200]}")
return False
return True
async def search_videos(
client: httpx.AsyncClient,
api_key: str,
query: str,
per_page: int = 5
) -> list[dict]:
"""Search for videos on Pexels."""
headers = {"Authorization": api_key}
params = {
"query": query,
"per_page": per_page,
"orientation": "landscape",
"size": "medium",
}
try:
response = await client.get(
f"{PEXELS_API_BASE}/search",
headers=headers,
params=params,
timeout=30.0
)
response.raise_for_status()
data = response.json()
return data.get("videos", [])
except httpx.HTTPError as e:
print(f" ⚠️ Error searching for '{query}': {e}")
return []
def get_best_video_file(video: dict, max_width: int = 1280) -> Optional[dict]:
"""Get the best quality video file under max_width."""
video_files = video.get("video_files", [])
# Filter to reasonable sizes and sort by quality
suitable = [
vf for vf in video_files
if vf.get("width", 0) <= max_width and vf.get("quality") in ("hd", "sd")
]
if not suitable:
suitable = video_files
if not suitable:
return None
# Sort by width descending (prefer higher quality)
suitable.sort(key=lambda x: x.get("width", 0), reverse=True)
return suitable[0]
async def download_video(
client: httpx.AsyncClient,
url: str,
dest_path: Path
) -> bool:
"""Download a video file."""
try:
async with client.stream("GET", url, timeout=120.0) as response:
response.raise_for_status()
with open(dest_path, "wb") as f:
async for chunk in response.aiter_bytes(chunk_size=8192):
f.write(chunk)
return True
except Exception as e:
print(f" ⚠️ Download error: {e}")
return False
def sanitize_metadata(metadata: dict) -> dict:
"""Sanitize metadata values to ASCII-only for S3."""
result = {}
for k, v in metadata.items():
# Convert to string and encode as ASCII, replacing non-ASCII chars
val = str(v).encode("ascii", errors="replace").decode("ascii")
result[k] = val
return result
async def upload_to_s3(
s3_client,
local_path: Path,
bucket: str,
s3_key: str,
metadata: dict
) -> bool:
"""Upload a file to S3 with metadata."""
try:
extra_args = {
"ContentType": "video/mp4",
"Metadata": sanitize_metadata(metadata)
}
await s3_client.upload_file(str(local_path), bucket, s3_key, ExtraArgs=extra_args)
return True
except ClientError as e:
print(f" ⚠️ S3 upload error: {e}")
return False
def generate_filename(video: dict, query: str, index: int) -> str:
"""Generate a descriptive filename for the video."""
clean_query = query.replace(" ", "_").replace("/", "-")[:30]
video_id = video.get("id", "unknown")
return f"{clean_query}_{video_id}_{index:02d}.mp4"
async def process_video(
http_client: httpx.AsyncClient,
s3_client,
video: dict,
index: int,
temp_dir: Path,
local_dir: Optional[Path],
bucket: Optional[str],
s3_prefix: str,
download_sem: asyncio.Semaphore,
upload_sem: asyncio.Semaphore,
normalize: bool = True,
normalize_width: int = NORMALIZE_WIDTH,
normalize_height: int = NORMALIZE_HEIGHT,
normalize_fps: int = NORMALIZE_FPS,
) -> Optional[dict]:
"""Download, normalize, and upload a single video."""
video_file = get_best_video_file(video)
if not video_file:
print(f" {index+1:2d}. ⚠️ No suitable video file found, skipping")
return None
filename = generate_filename(video, video["_query"], index)
raw_path = temp_dir / f"raw_{filename}"
normalized_path = temp_dir / filename
print(f" {index+1:2d}. Downloading: {filename}")
download_url = video_file.get("link")
if not download_url:
print(f" ⚠️ No download URL, skipping")
return None
# Download with semaphore
async with download_sem:
if not await download_video(http_client, download_url, raw_path):
return None
raw_size_mb = raw_path.stat().st_size / (1024 * 1024)
print(f" ✅ Downloaded ({raw_size_mb:.1f} MB)")
# Normalize video
if normalize:
print(f" 🔄 Normalizing to {normalize_width}x{normalize_height}@{normalize_fps}fps...")
# Run normalization in thread pool to not block event loop
success = await asyncio.to_thread(
normalize_video,
raw_path,
normalized_path,
normalize_width,
normalize_height,
normalize_fps,
)
if not success:
print(f" ⚠️ Normalization failed, using original")
shutil.move(raw_path, normalized_path)
else:
normalized_size_mb = normalized_path.stat().st_size / (1024 * 1024)
print(f" ✅ Normalized ({raw_size_mb:.1f} MB → {normalized_size_mb:.1f} MB)")
raw_path.unlink(missing_ok=True)
# Update dimensions to normalized values
final_width = normalize_width
final_height = normalize_height
else:
# No normalization, just rename
shutil.move(raw_path, normalized_path)
final_width = video_file.get("width")
final_height = video_file.get("height")
# Copy to local dir if specified
if local_dir:
local_path = local_dir / filename
shutil.copy2(normalized_path, local_path)
print(f" 📁 Saved locally: {local_path}")
result = None
# Upload to S3
if s3_client and bucket:
s3_key = f"{s3_prefix}{filename}"
metadata = {
"pexels_id": str(video.get("id", "")),
"query": video["_query"],
"width": str(final_width),
"height": str(final_height),
"duration": str(video.get("duration", "")),
"photographer": video.get("user", {}).get("name", ""),
"normalized": str(normalize),
}
async with upload_sem:
if await upload_to_s3(s3_client, normalized_path, bucket, s3_key, metadata):
print(f" ☁️ Uploaded to: s3://{bucket}/{s3_key}")
result = {
"filename": filename,
"s3_uri": f"s3://{bucket}/{s3_key}",
"s3_key": s3_key,
"pexels_id": video.get("id"),
"query": video["_query"],
"duration": video.get("duration"),
"width": final_width,
"height": final_height,
}
elif local_dir:
# Local only mode
result = {
"filename": filename,
"local_path": str(local_dir / filename),
"pexels_id": video.get("id"),
"query": video["_query"],
"duration": video.get("duration"),
"width": final_width,
"height": final_height,
}
# Clean up temp file
normalized_path.unlink(missing_ok=True)
return result
async def download_sample_videos(
api_key: str | None = None,
bucket: str | None = None,
total: int = 20,
per_query: int = 1,
local_dir: str | None = None,
dry_run: bool = False,
skip_s3: bool = False,
normalize: bool = True,
width: int = NORMALIZE_WIDTH,
height: int = NORMALIZE_HEIGHT,
fps: int = NORMALIZE_FPS,
s3_prefix: str = S3_VIDEOS_PREFIX,
overwrite: bool = True,
) -> list[str]:
"""Download videos from Pexels, normalize them, and upload to S3.
If a manifest already exists in S3, returns the existing video paths
without downloading new videos.
Args:
api_key: Pexels API key. Falls back to PEXELS_API_KEY env var.
bucket: S3 bucket name. Falls back to S3_BUCKET env var.
total: Total number of videos to download.
per_query: Number of videos per search query.
local_dir: Optional local directory to save videos.
dry_run: If True, only show what would be downloaded.
skip_s3: If True, skip S3 upload (local download only).
normalize: If True, normalize videos to consistent specs.
width: Normalized video width.
height: Normalized video height.
fps: Normalized video FPS.
Returns:
List of video paths (S3 URIs or local paths).
"""
# Get configuration from args or environment
api_key = api_key or os.environ.get("PEXELS_API_KEY")
bucket = bucket or os.environ.get("S3_BUCKET")
if not bucket and not skip_s3:
print("❌ Error: S3 bucket required (--bucket or S3_BUCKET)")
print(" Set it or use --skip-s3 to only download locally")
sys.exit(1)
# Setup S3 session
session = aioboto3.Session(region_name=get_s3_region(bucket))
s3_client = None
if not skip_s3:
async with session.client("s3") as s3:
try:
await s3.head_bucket(Bucket=bucket)
print(f"✅ S3 bucket '{bucket}' accessible")
except ClientError as e:
print(f"❌ Error: Cannot access S3 bucket '{bucket}': {e}")
sys.exit(1)
if overwrite:
print("🔄 Overwriting existing manifest")
await s3.delete_object(Bucket=bucket, Key=f"{s3_prefix}manifest.json")
# Check if manifest already exists - return early if so
manifest_key = f"{s3_prefix}manifest.json"
try:
response = await s3.get_object(Bucket=bucket, Key=manifest_key)
manifest_data = await response["Body"].read()
manifest = json.loads(manifest_data.decode("utf-8"))
# Extract paths from existing manifest
video_paths = []
for v in manifest.get("videos", []):
if "s3_uri" in v:
video_paths.append(v["s3_uri"])
elif "local_path" in v:
video_paths.append(v["local_path"])
print(f"✅ Found existing manifest with {len(video_paths)} videos in S3")
print(f" Skipping Pexels API download")
return video_paths
except ClientError as e:
if e.response["Error"]["Code"] != "NoSuchKey":
raise
# Manifest doesn't exist, continue with download
print("📥 No existing manifest found, will download from Pexels")
# Need API key for downloading
if not api_key:
print("❌ Error: Pexels API key required (--api-key or PEXELS_API_KEY)")
print(" Get your free API key at: https://www.pexels.com/api/")
sys.exit(1)
# Create local directory if specified
local_dir_path = None
if local_dir:
local_dir_path = Path(local_dir)
local_dir_path.mkdir(parents=True, exist_ok=True)
print(f"📁 Local directory: {local_dir_path}")
# Create temp directory for downloads
temp_dir = Path(tempfile.mkdtemp(prefix="pexels_videos_"))
print(f"📁 Temp directory: {temp_dir}")
# Track downloaded videos
video_ids_seen = set()
print(f"\n🔍 Searching for {total} videos across {len(SEARCH_QUERIES)} queries...\n")
# Search and collect videos concurrently
all_videos = []
async with httpx.AsyncClient() as http_client:
# Search all queries concurrently
search_tasks = [
search_videos(http_client, api_key, query, per_page=per_query + 2)
for query in SEARCH_QUERIES
]
results = await asyncio.gather(*search_tasks)
for query, videos in zip(SEARCH_QUERIES, results):
print(f" Found {len(videos)} for '{query}'")
for video in videos:
if len(all_videos) >= total:
break
video_id = video.get("id")
if video_id and video_id not in video_ids_seen:
video_ids_seen.add(video_id)
video["_query"] = query
all_videos.append(video)
all_videos = all_videos[:total]
print(f"\n📹 Selected {len(all_videos)} unique videos\n")
if dry_run:
print("🔍 DRY RUN - Would download these videos:\n")
for i, video in enumerate(all_videos):
video_file = get_best_video_file(video)
if video_file:
filename = generate_filename(video, video["_query"], i)
print(f" {i+1:2d}. {filename}")
print(f" URL: {video_file.get('link', 'N/A')[:80]}...")
print(f" Size: {video_file.get('width')}x{video_file.get('height')}")
print()
return []
# Download and upload videos concurrently
if normalize:
print(f"⬇️ Downloading, normalizing ({width}x{height}@{fps}fps), and uploading videos...\n")
else:
print("⬇️ Downloading and uploading videos (no normalization)...\n")
download_sem = asyncio.Semaphore(MAX_CONCURRENT_DOWNLOADS)
upload_sem = asyncio.Semaphore(MAX_CONCURRENT_UPLOADS)
downloaded_videos = []
async with httpx.AsyncClient() as http_client:
if skip_s3:
# No S3, just download
tasks = [
process_video(
http_client, None, video, i, temp_dir, local_dir_path,
None, s3_prefix, download_sem, upload_sem,
normalize=normalize,
normalize_width=width,
normalize_height=height,
normalize_fps=fps,
)
for i, video in enumerate(all_videos)
]
results = await asyncio.gather(*tasks, return_exceptions=True)
downloaded_videos = [r for r in results if r is not None and not isinstance(r, Exception)]
for r in results:
if isinstance(r, Exception):
print(f" ⚠️ Task failed: {r}")
else:
# With S3
async with session.client("s3") as s3_client:
tasks = [
process_video(
http_client, s3_client, video, i, temp_dir, local_dir_path,
bucket, s3_prefix, download_sem, upload_sem,
normalize=normalize,
normalize_width=width,
normalize_height=height,
normalize_fps=fps,
)
for i, video in enumerate(all_videos)
]
results = await asyncio.gather(*tasks, return_exceptions=True)
downloaded_videos = [r for r in results if r is not None and not isinstance(r, Exception)]
# Log any exceptions
for r in results:
if isinstance(r, Exception):
print(f" ⚠️ Task failed: {r}")
# Save manifest
manifest = {
"total_videos": len(downloaded_videos),
"s3_bucket": bucket if not skip_s3 else None,
"s3_prefix": s3_prefix if not skip_s3 else None,
"local_dir": str(local_dir_path) if local_dir_path else None,
"normalized": normalize,
"normalize_settings": {
"width": width,
"height": height,
"fps": fps,
} if normalize else None,
"videos": downloaded_videos,
}
manifest_path = Path("video_manifest.json")
with open(manifest_path, "w") as f:
json.dump(manifest, f, indent=2)
print(f"\n📋 Manifest saved to: {manifest_path}")
# Also upload manifest to S3
if not skip_s3 and bucket:
async with session.client("s3") as s3_client:
manifest_s3_key = f"{s3_prefix}manifest.json"
try:
await s3_client.put_object(
Bucket=bucket,
Key=manifest_s3_key,
Body=json.dumps(manifest, indent=2),
ContentType="application/json"
)
print(f"☁️ Manifest uploaded to: s3://{bucket}/{manifest_s3_key}")
except ClientError as e:
print(f"⚠️ Failed to upload manifest: {e}")
# Cleanup temp dir
try:
temp_dir.rmdir()
except OSError:
pass # May not be empty if some downloads failed
print(f"\n✅ Done! Processed {len(downloaded_videos)} videos.")
# Extract paths from downloaded videos
video_paths = []
for v in downloaded_videos:
if "s3_uri" in v:
video_paths.append(v["s3_uri"])
elif "local_path" in v:
video_paths.append(v["local_path"])
if video_paths:
print("\n📝 Sample paths for testing:")
for path in video_paths[:5]:
print(f" {path}")
return video_paths
def main():
parser = argparse.ArgumentParser(
description="Download Pexels videos, normalize them, and upload to S3",
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog="""
Normalization applies:
- Resolution scaling with letterboxing (preserves aspect ratio)
- Consistent FPS
- H.264 codec (libx264, main profile)
- 1-second GOP for fast seeking
- Removes audio
Examples:
# Download and normalize to default 1280x720@30fps
python scripts/download_stock_videos.py --api-key KEY --bucket BUCKET
# Custom resolution
python scripts/download_stock_videos.py --api-key KEY --bucket BUCKET --width 1920 --height 1080 --fps 24
# Skip normalization (upload original files)
python scripts/download_stock_videos.py --api-key KEY --bucket BUCKET --no-normalize
"""
)
parser.add_argument("--api-key", type=str, help="Pexels API key (or set PEXELS_API_KEY)")
parser.add_argument("--bucket", type=str, help="S3 bucket name (or set S3_BUCKET)")
parser.add_argument("--total", type=int, default=20, help="Total videos to download")
parser.add_argument("--per-query", type=int, default=1, help="Videos per search query")
parser.add_argument("--local-dir", type=str, help="Also save videos locally to this directory")
parser.add_argument("--dry-run", action="store_true", help="Just show what would be downloaded")
parser.add_argument("--skip-s3", action="store_true", help="Skip S3 upload, only download locally")
# Normalization options
parser.add_argument("--no-normalize", action="store_true",
help="Skip video normalization (upload original files)")
parser.add_argument("--width", type=int, default=NORMALIZE_WIDTH,
help=f"Normalized video width (default: {NORMALIZE_WIDTH})")
parser.add_argument("--height", type=int, default=NORMALIZE_HEIGHT,
help=f"Normalized video height (default: {NORMALIZE_HEIGHT})")
parser.add_argument("--fps", type=int, default=NORMALIZE_FPS,
help=f"Normalized video FPS (default: {NORMALIZE_FPS})")
args = parser.parse_args()
asyncio.run(download_sample_videos(
api_key=args.api_key,
bucket=args.bucket,
total=args.total,
per_query=args.per_query,
local_dir=args.local_dir,
dry_run=args.dry_run,
skip_s3=args.skip_s3,
normalize=not args.no_normalize,
width=args.width,
height=args.height,
fps=args.fps,
))
if __name__ == "__main__":
main()
doc/source/serve/tutorials/video-analysis/services.yaml
+98
View File
@@ -0,0 +1,98 @@
# View the docs https://docs.anyscale.com/reference/service-api#serviceconfig.
name: video-analysis-app
image_uri: # add image uri here
compute_config:
head_node:
instance_type: m5.2xlarge
worker_nodes:
- instance_type: m8i.4xlarge
min_nodes: 1
max_nodes: 32
- instance_type: g6.xlarge
min_nodes: 1
max_nodes: 32
working_dir: .
excludes:
- .git
- .env
- .venv
- '**/*.egg-info/**'
- '**/.DS_Store/**'
- '**/__pycache__/**'
applications:
- name: app1
route_prefix: /
import_path: app:app
runtime_env: {}
autoscaling_policy:
policy_function: autoscaling_policy:coordinated_scaling_policy
deployments:
- name: VideoEncoder
num_replicas: "auto"
max_ongoing_requests: 2
autoscaling_config:
min_replicas: 1
initial_replicas: null
max_replicas: 32
target_ongoing_requests: 2.0
metrics_interval_s: 10.0
look_back_period_s: 30.0
smoothing_factor: 1.0
upscale_smoothing_factor: null
downscale_smoothing_factor: null
upscaling_factor: null
downscaling_factor: null
downscale_delay_s: 600.0
downscale_to_zero_delay_s: null
upscale_delay_s: 30.0
aggregation_function: mean
ray_actor_options:
num_cpus: 2.0
num_gpus: 1.0
- name: MultiDecoder
num_replicas: "auto"
max_ongoing_requests: 4
autoscaling_config:
min_replicas: 1
initial_replicas: null
max_replicas: 32
target_ongoing_requests: 2.0
metrics_interval_s: 10.0
look_back_period_s: 30.0
smoothing_factor: 1.0
upscale_smoothing_factor: null
downscale_smoothing_factor: null
upscaling_factor: null
downscaling_factor: null
downscale_delay_s: 600.0
downscale_to_zero_delay_s: null
upscale_delay_s: 30.0
aggregation_function: mean
ray_actor_options:
num_cpus: 1.0
- name: VideoAnalyzer
num_replicas: "auto"
max_ongoing_requests: 4
autoscaling_config:
min_replicas: 1
initial_replicas: null
max_replicas: 64
target_ongoing_requests: 2.0
metrics_interval_s: 10.0
look_back_period_s: 30.0
smoothing_factor: 1.0
upscale_smoothing_factor: null
downscale_smoothing_factor: null
upscaling_factor: null
downscaling_factor: null
downscale_delay_s: 600.0
downscale_to_zero_delay_s: null
upscale_delay_s: 30.0
aggregation_function: mean
ray_actor_options:
num_cpus: 6.0
doc/source/serve/tutorials/video-analysis/textbanks/__init__.py
+5
View File
@@ -0,0 +1,5 @@
from .tags import TAGS
from .descriptions import DESCRIPTIONS
__all__ = ["TAGS", "DESCRIPTIONS"]
doc/source/serve/tutorials/video-analysis/textbanks/descriptions.py
+45
View File
@@ -0,0 +1,45 @@
# Descriptions for caption retrieval
DESCRIPTIONS = [
"A person cooking in a kitchen",
"Someone preparing food on a counter",
"A chef working in a professional kitchen",
"People eating at a dining table",
"A group having a meal together",
"A person working at a desk",
"Someone typing on a laptop",
"A business meeting in progress",
"A presentation being given",
"People collaborating in an office",
"A teacher lecturing in a classroom",
"Students sitting at desks",
"A person giving a speech",
"Someone writing on a whiteboard",
"A customer shopping in a store",
"People browsing products on shelves",
"A cashier at a checkout counter",
"A person exercising at a gym",
"Someone lifting weights",
"A person running on a treadmill",
"People walking on a city sidewalk",
"Pedestrians crossing a street",
"Traffic moving through an intersection",
"Cars driving on a road",
"A vehicle parked in a lot",
"People walking through a park",
"Someone jogging outdoors",
"A group having a conversation",
"Two people talking face to face",
"A person on a phone call",
"Someone reading a book",
"A person watching television",
"People waiting in line",
"A crowded public space",
"An empty hallway or corridor",
"A person entering a building",
"Someone opening a door",
"A delivery being made",
"A person carrying boxes",
"Workers in a warehouse",
]
doc/source/serve/tutorials/video-analysis/textbanks/tags.py
+24
View File
@@ -0,0 +1,24 @@
# Tags for zero-shot scene classification (short category labels)
TAGS = [
"kitchen",
"living room",
"office",
"meeting room",
"classroom",
"restaurant",
"cafe",
"grocery store",
"gym",
"warehouse",
"parking lot",
"city street",
"park",
"shopping mall",
"beach",
"sports field",
"hallway",
"lobby",
"bathroom",
"bedroom",
]
doc/source/serve/tutorials/video-analysis/utils/__init__.py
View File
doc/source/serve/tutorials/video-analysis/utils/s3.py
+8
View File
@@ -0,0 +1,8 @@
import boto3
def get_s3_region(bucket: str) -> str:
"""Get the region of the S3 bucket"""
s3 = boto3.client("s3")
response = s3.get_bucket_location(Bucket=bucket)
# AWS returns None for us-east-1, otherwise returns the region name
return response["LocationConstraint"] or "us-east-1"
doc/source/serve/tutorials/video-analysis/utils/video.py
+473
View File
@@ -0,0 +1,473 @@
"""Video loading and frame sampling utilities using ffmpeg."""
import asyncio
import json
import subprocess
from dataclasses import dataclass
from typing import Optional
import numpy as np
from PIL import Image
from constants import NUM_WORKERS
@dataclass
class VideoMetadata:
"""Video metadata extracted from ffprobe."""
duration: float # seconds
fps: float
width: int
height: int
num_frames: int
def get_video_metadata(video_path: str) -> VideoMetadata:
"""Get video metadata using ffprobe. Works with local files and URLs."""
# Use JSON output for reliable field parsing (CSV order is unpredictable)
cmd = [
"ffprobe",
"-v", "error",
"-select_streams", "v:0",
"-show_entries", "stream=width,height,r_frame_rate,nb_frames,duration",
"-of", "json",
video_path,
]
result = subprocess.run(cmd, capture_output=True, text=True, check=True)
data = json.loads(result.stdout)
stream = data["streams"][0]
width = int(stream["width"])
height = int(stream["height"])
# Parse frame rate (can be "30/1" or "29.97")
fps_str = stream["r_frame_rate"]
if "/" in fps_str:
num, den = fps_str.split("/")
fps = float(num) / float(den)
else:
fps = float(fps_str)
# nb_frames might be N/A for some formats
try:
num_frames = int(stream.get("nb_frames", 0))
except (ValueError, TypeError):
num_frames = 0
# Duration might be in stream or need to be fetched from format
try:
duration = float(stream.get("duration", 0))
except (ValueError, TypeError):
duration = 0
if duration == 0:
# Fallback: get duration from format
cmd2 = [
"ffprobe",
"-v", "error",
"-show_entries", "format=duration",
"-of", "json",
video_path,
]
result2 = subprocess.run(cmd2, capture_output=True, text=True, check=True)
data2 = json.loads(result2.stdout)
duration = float(data2["format"]["duration"])
if num_frames == 0:
num_frames = int(duration * fps)
return VideoMetadata(
duration=duration,
fps=fps,
width=width,
height=height,
num_frames=num_frames,
)
def extract_frames_ffmpeg(
video_path: str,
start_time: float,
duration: float,
num_frames: int,
target_size: int = 384,
ffmpeg_threads: int = 0,
) -> np.ndarray:
"""
Extract frames from a video segment using ffmpeg.
Works with local files and URLs (including presigned S3 URLs).
Args:
video_path: Path to video file or URL
start_time: Start time in seconds
duration: Duration to extract in seconds
num_frames: Number of frames to extract (uniformly sampled)
target_size: Output frame size (square)
ffmpeg_threads: Number of threads for FFmpeg (0 = auto)
Returns:
np.ndarray of shape (num_frames, target_size, target_size, 3) uint8 RGB
"""
# Calculate output fps to get exactly num_frames
output_fps = num_frames / duration if duration > 0 else num_frames
cmd = [
"ffmpeg",
"-threads", str(ffmpeg_threads),
"-ss", str(start_time),
"-t", str(duration),
"-i", video_path,
"-vf", f"fps={output_fps},scale={target_size}:{target_size}",
"-pix_fmt", "rgb24",
"-f", "rawvideo",
"-",
]
result = subprocess.run(
cmd,
capture_output=True,
check=True,
)
# Parse raw video frames
frame_size = target_size * target_size * 3
raw_data = result.stdout
actual_frames = len(raw_data) // frame_size
if actual_frames == 0:
raise ValueError(f"No frames extracted from {video_path} at {start_time}s")
frames = np.frombuffer(raw_data[:actual_frames * frame_size], dtype=np.uint8)
frames = frames.reshape(actual_frames, target_size, target_size, 3)
# Pad or truncate to exact num_frames
if len(frames) < num_frames:
# Pad by repeating last frame
padding = np.tile(frames[-1:], (num_frames - len(frames), 1, 1, 1))
frames = np.concatenate([frames, padding], axis=0)
elif len(frames) > num_frames:
frames = frames[:num_frames]
return frames
@dataclass
class VideoChunk:
"""Represents a chunk of video to process."""
index: int
start_time: float
duration: float
frames: Optional[np.ndarray] = None
async def extract_frames_async(
video_path: str,
start_time: float,
duration: float,
num_frames: int,
target_size: int = 384,
ffmpeg_threads: int = 0,
) -> np.ndarray:
"""Async wrapper for extract_frames_ffmpeg using thread pool."""
return await asyncio.to_thread(
extract_frames_ffmpeg,
video_path,
start_time,
duration,
num_frames,
target_size,
ffmpeg_threads,
)
def _extract_all_chunks_single_ffmpeg(
video_path: str,
chunk_defs: list[tuple[int, float, float]],
num_frames_per_chunk: int,
target_size: int,
ffmpeg_threads: int = 0,
) -> list[np.ndarray]:
"""
Extract frames for ALL chunks in a single FFmpeg call.
Uses the select filter to pick specific frame timestamps, avoiding
multiple process spawns and file seeks.
Args:
video_path: Path to video file or URL
chunk_defs: List of (index, start_time, duration) tuples
num_frames_per_chunk: Frames to extract per chunk
target_size: Output frame size (square)
ffmpeg_threads: Number of threads for FFmpeg (0 = auto)
Returns:
List of numpy arrays, one per chunk
"""
# Build list of all timestamps to extract
all_timestamps = []
for idx, start, duration in chunk_defs:
# Uniformly sample timestamps within each chunk
for i in range(num_frames_per_chunk):
t = start + (i * duration / num_frames_per_chunk)
all_timestamps.append(t)
if not all_timestamps:
return []
# Build select filter expression: select frames nearest to our timestamps
# Using eq(n,frame_num) would require knowing frame numbers, so instead
# we use pts-based selection with a small tolerance
# The 'select' filter with 'lt(prev_pts,T)*gte(pts,T)' picks first frame >= T
# For efficiency, we'll extract at a high fps and pick specific frames,
# or use the thumbnail filter. But simplest: extract all frames near our
# timestamps using the 'select' filter.
# Build the select expression for all timestamps
# select='eq(n,0)+eq(n,10)+eq(n,20)...' but we need PTS-based selection
# Better approach: use fps filter to get enough frames, then select in numpy
# Calculate total time span and required fps
min_t = min(all_timestamps)
max_t = max(all_timestamps)
total_duration = max_t - min_t + 0.1 # small buffer
# We need at least len(all_timestamps) frames over total_duration
# But we want to be precise, so let's use select filter with expressions
# Build select expression: for each timestamp T, select frame where pts >= T and prev_pts < T
# This is complex. Simpler approach: output frames at specific PTS values.
# Most efficient single-pass approach: use the 'select' filter with timestamp checks
# select='between(t,T1-eps,T1+eps)+between(t,T2-eps,T2+eps)+...'
eps = 0.02 # 20ms tolerance for frame selection
select_parts = [f"between(t,{t-eps},{t+eps})" for t in all_timestamps]
select_expr = "+".join(select_parts)
cmd = [
"ffmpeg",
"-threads", str(ffmpeg_threads),
"-i", video_path,
"-vf", f"select='{select_expr}',scale={target_size}:{target_size}",
"-vsync", "vfr", # Variable frame rate to preserve selected frames
"-pix_fmt", "rgb24",
"-f", "rawvideo",
"-",
]
result = subprocess.run(cmd, capture_output=True, check=True)
# Parse raw video frames
frame_size = target_size * target_size * 3
raw_data = result.stdout
total_frames = len(raw_data) // frame_size
if total_frames == 0:
raise ValueError(f"No frames extracted from {video_path}")
all_frames = np.frombuffer(raw_data[:total_frames * frame_size], dtype=np.uint8)
all_frames = all_frames.reshape(total_frames, target_size, target_size, 3)
# Split into chunks
chunk_frames = []
frame_idx = 0
for idx, start, duration in chunk_defs:
# Take num_frames_per_chunk frames for this chunk
end_idx = min(frame_idx + num_frames_per_chunk, total_frames)
chunk_data = all_frames[frame_idx:end_idx]
# Pad if needed
if len(chunk_data) < num_frames_per_chunk:
if len(chunk_data) == 0:
# No frames for this chunk, create black frames
chunk_data = np.zeros((num_frames_per_chunk, target_size, target_size, 3), dtype=np.uint8)
else:
padding = np.tile(chunk_data[-1:], (num_frames_per_chunk - len(chunk_data), 1, 1, 1))
chunk_data = np.concatenate([chunk_data, padding], axis=0)
chunk_frames.append(chunk_data)
frame_idx = end_idx
return chunk_frames
async def chunk_video_async(
video_path: str,
chunk_duration: float = 10.0,
num_frames_per_chunk: int = 16,
target_size: int = 384,
use_single_ffmpeg: bool = False,
ffmpeg_threads: int = 0,
) -> list[VideoChunk]:
"""
Split video into fixed-duration chunks with frame extraction.
Works with local files and URLs (including presigned S3 URLs).
Args:
video_path: Path to video file or URL
chunk_duration: Duration of each chunk in seconds
num_frames_per_chunk: Frames to extract per chunk
target_size: Frame size
use_single_ffmpeg: If True, extract all chunks in one FFmpeg call (faster).
If False, use parallel FFmpeg calls per chunk.
ffmpeg_threads: Number of threads for FFmpeg decoding (0 = auto)
Returns:
List of VideoChunk with frames loaded
"""
# Get metadata (sync call, fast)
metadata = await asyncio.to_thread(get_video_metadata, video_path)
# Build chunk definitions
chunk_defs = []
start = 0.0
index = 0
while start < metadata.duration:
duration = min(chunk_duration, metadata.duration - start)
# Skip very short final chunks
if duration < 0.5:
break
chunk_defs.append((index, start, duration))
start += chunk_duration
index += 1
if not chunk_defs:
return []
if use_single_ffmpeg:
# Single FFmpeg call - more efficient, especially for URLs
frame_results = await asyncio.to_thread(
_extract_all_chunks_single_ffmpeg,
video_path,
chunk_defs,
num_frames_per_chunk,
target_size,
ffmpeg_threads,
)
else:
# Multiple parallel FFmpeg calls, limited to NUM_WORKERS concurrency
semaphore = asyncio.Semaphore(NUM_WORKERS)
async def extract_with_limit(idx, start, duration):
async with semaphore:
return await extract_frames_async(
video_path,
start_time=start,
duration=duration,
num_frames=num_frames_per_chunk,
target_size=target_size,
ffmpeg_threads=ffmpeg_threads,
)
extraction_tasks = [
extract_with_limit(idx, start, duration)
for idx, start, duration in chunk_defs
]
frame_results = await asyncio.gather(*extraction_tasks)
# Build chunk objects
chunks = [
VideoChunk(
index=idx,
start_time=start,
duration=duration,
frames=frames,
)
for (idx, start, duration), frames in zip(chunk_defs, frame_results)
]
return chunks
def chunk_video(
video_path: str,
chunk_duration: float = 10.0,
num_frames_per_chunk: int = 16,
target_size: int = 384,
use_single_ffmpeg: bool = True,
ffmpeg_threads: int = 0,
) -> list[VideoChunk]:
"""
Split video into fixed-duration chunks.
Args:
video_path: Path to video file or URL
chunk_duration: Duration of each chunk in seconds
num_frames_per_chunk: Frames to extract per chunk
target_size: Frame size
use_single_ffmpeg: If True, extract all chunks in one FFmpeg call (faster).
If False, use sequential FFmpeg calls per chunk.
ffmpeg_threads: Number of threads for FFmpeg decoding (0 = auto)
Returns:
List of VideoChunk with frames loaded
"""
metadata = get_video_metadata(video_path)
# Build chunk definitions
chunk_defs = []
start = 0.0
index = 0
while start < metadata.duration:
duration = min(chunk_duration, metadata.duration - start)
# Skip very short final chunks
if duration < 0.5:
break
chunk_defs.append((index, start, duration))
start += chunk_duration
index += 1
if not chunk_defs:
return []
if use_single_ffmpeg:
# Single FFmpeg call - more efficient, especially for URLs
frame_results = _extract_all_chunks_single_ffmpeg(
video_path,
chunk_defs,
num_frames_per_chunk,
target_size,
ffmpeg_threads,
)
else:
# Sequential FFmpeg calls (original approach)
frame_results = []
for idx, start, duration in chunk_defs:
frames = extract_frames_ffmpeg(
video_path,
start_time=start,
duration=duration,
num_frames=num_frames_per_chunk,
target_size=target_size,
ffmpeg_threads=ffmpeg_threads,
)
frame_results.append(frames)
# Build chunk objects
chunks = [
VideoChunk(
index=idx,
start_time=start,
duration=duration,
frames=frames,
)
for (idx, start, duration), frames in zip(chunk_defs, frame_results)
]
return chunks
def frames_to_pil_list(frames: np.ndarray) -> list[Image.Image]:
"""Convert numpy frames array to list of PIL Images."""
return [Image.fromarray(frame) for frame in frames]