wehub/ray-project--ray
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 13:17:40 +08:00
commit f1825c8ceb
10096 changed files with 2364182 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/source/serve/tutorials/aws-neuron-core-inference-stable-diffusion.md
+121
View File
@@ -0,0 +1,121 @@
---
orphan: true
---
# Serve an Inference with Stable Diffusion Model on AWS NeuronCores Using FastAPI
This example uses a precompiled Stable Diffusion XL model and deploys on an AWS Inferentia2 (Inf2) instance using Ray Serve and FastAPI.
:::{note}
Before starting this example:
* Set up [PyTorch Neuron](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/general/setup/torch-neuronx.html#setup-torch-neuronx)
* Install AWS NeuronCore drivers and tools, and torch-neuronx based on the instance-type
:::
```bash
pip install "optimum-neuron==0.0.13" "diffusers==0.21.4"
pip install "ray[serve]" requests transformers
```
This example uses the [Stable Diffusion-XL](https://huggingface.co/aws-neuron/stable-diffusion-xl-base-1-0-1024x1024) model and [FastAPI](https://fastapi.tiangolo.com/). This model is compiled with AWS Neuron and is ready to run inference. However, you can choose a different Stable Diffusion model and compile it to be compatible for running inference on AWS Inferentia2 instances.
The model in this example is ready for deployment. Save the following code to a file named aws_neuron_core_inference_serve_stable_diffusion.py.
Use `serve run aws_neuron_core_inference_serve_stable_diffusion:entrypoint` to start the Serve application.
```{literalinclude} ../doc_code/aws_neuron_core_inference_serve_stable_diffusion.py
:language: python
:start-after: __neuron_serve_code_start__
:end-before: __neuron_serve_code_end__
```
You should see the following log messages when a deployment using RayServe is successful:
```text
2024-02-07 17:53:28,299 INFO worker.py:1715 -- Started a local Ray instance. View the dashboard at http://127.0.0.1:8265
(ProxyActor pid=25282) INFO 2024-02-07 17:53:31,751 proxy 172.31.10.188 proxy.py:1128 - Proxy actor fd464602af1e456162edf6f901000000 starting on node 5a8e0c24b22976f1f7672cc54f13ace25af3664a51429d8e332c0679.
(ProxyActor pid=25282) INFO 2024-02-07 17:53:31,755 proxy 172.31.10.188 proxy.py:1333 - Starting HTTP server on node: 5a8e0c24b22976f1f7672cc54f13ace25af3664a51429d8e332c0679 listening on port 8000
(ProxyActor pid=25282) INFO: Started server process [25282]
(ServeController pid=25233) INFO 2024-02-07 17:53:31,921 controller 25233 deployment_state.py:1545 - Deploying new version of deployment StableDiffusionV2 in application 'default'. Setting initial target number of replicas to 1.
(ServeController pid=25233) INFO 2024-02-07 17:53:31,922 controller 25233 deployment_state.py:1545 - Deploying new version of deployment APIIngress in application 'default'. Setting initial target number of replicas to 1.
(ServeController pid=25233) INFO 2024-02-07 17:53:32,024 controller 25233 deployment_state.py:1829 - Adding 1 replica to deployment StableDiffusionV2 in application 'default'.
(ServeController pid=25233) INFO 2024-02-07 17:53:32,029 controller 25233 deployment_state.py:1829 - Adding 1 replica to deployment APIIngress in application 'default'.
Fetching 20 files: 100%|██████████| 20/20 [00:00<00:00, 195538.65it/s]
(ServeController pid=25233) WARNING 2024-02-07 17:54:02,114 controller 25233 deployment_state.py:2171 - Deployment 'StableDiffusionV2' in application 'default' has 1 replicas that have taken more than 30s to initialize. This may be caused by a slow __init__ or reconfigure method.
(ServeController pid=25233) WARNING 2024-02-07 17:54:32,170 controller 25233 deployment_state.py:2171 - Deployment 'StableDiffusionV2' in application 'default' has 1 replicas that have taken more than 30s to initialize. This may be caused by a slow __init__ or reconfigure method.
(ServeController pid=25233) WARNING 2024-02-07 17:55:02,344 controller 25233 deployment_state.py:2171 - Deployment 'StableDiffusionV2' in application 'default' has 1 replicas that have taken more than 30s to initialize. This may be caused by a slow __init__ or reconfigure method.
(ServeController pid=25233) WARNING 2024-02-07 17:55:32,418 controller 25233 deployment_state.py:2171 - Deployment 'StableDiffusionV2' in application 'default' has 1 replicas that have taken more than 30s to initialize. This may be caused by a slow __init__ or reconfigure method.
2024-02-07 17:55:46,263 SUCC scripts.py:483 -- Deployed Serve app successfully.
```
Use the following code to send requests:
```python
import requests
prompt = "a zebra is dancing in the grass, river, sunlit"
input = "%20".join(prompt.split(" "))
resp = requests.get(f"http://127.0.0.1:8000/imagine?prompt={input}")
print("Write the response to `output.png`.")
with open("output.png", "wb") as f:
f.write(resp.content)
```
You should see the following log messages when a request is sent to the endpoint:
```text
(ServeReplica:default:StableDiffusionV2 pid=25320) Prompt: a zebra is dancing in the grass, river, sunlit
0%| | 0/50 [00:00<?, ?it/s]2 pid=25320)
2%|▏ | 1/50 [00:00<00:14, 3.43it/s]320)
4%|▍ | 2/50 [00:00<00:13, 3.62it/s]320)
6%|▌ | 3/50 [00:00<00:12, 3.73it/s]320)
8%|▊ | 4/50 [00:01<00:12, 3.78it/s]320)
10%|█ | 5/50 [00:01<00:11, 3.81it/s]320)
12%|█▏ | 6/50 [00:01<00:11, 3.82it/s]320)
14%|█▍ | 7/50 [00:01<00:11, 3.83it/s]320)
16%|█▌ | 8/50 [00:02<00:10, 3.84it/s]320)
18%|█▊ | 9/50 [00:02<00:10, 3.85it/s]320)
20%|██ | 10/50 [00:02<00:10, 3.85it/s]20)
22%|██▏ | 11/50 [00:02<00:10, 3.85it/s]20)
24%|██▍ | 12/50 [00:03<00:09, 3.86it/s]20)
26%|██▌ | 13/50 [00:03<00:09, 3.86it/s]20)
28%|██▊ | 14/50 [00:03<00:09, 3.85it/s]20)
30%|███ | 15/50 [00:03<00:09, 3.85it/s]20)
32%|███▏ | 16/50 [00:04<00:08, 3.85it/s]20)
34%|███▍ | 17/50 [00:04<00:08, 3.85it/s]20)
36%|███▌ | 18/50 [00:04<00:08, 3.85it/s]20)
38%|███▊ | 19/50 [00:04<00:08, 3.86it/s]20)
40%|████ | 20/50 [00:05<00:07, 3.85it/s]20)
42%|████▏ | 21/50 [00:05<00:07, 3.85it/s]20)
44%|████▍ | 22/50 [00:05<00:07, 3.85it/s]20)
46%|████▌ | 23/50 [00:06<00:07, 3.81it/s]20)
48%|████▊ | 24/50 [00:06<00:06, 3.81it/s]20)
50%|█████ | 25/50 [00:06<00:06, 3.82it/s]20)
52%|█████▏ | 26/50 [00:06<00:06, 3.83it/s]20)
54%|█████▍ | 27/50 [00:07<00:05, 3.84it/s]20)
56%|█████▌ | 28/50 [00:07<00:05, 3.84it/s]20)
58%|█████▊ | 29/50 [00:07<00:05, 3.84it/s]20)
60%|██████ | 30/50 [00:07<00:05, 3.84it/s]20)
62%|██████▏ | 31/50 [00:08<00:04, 3.84it/s]20)
64%|██████▍ | 32/50 [00:08<00:04, 3.84it/s]20)
66%|██████▌ | 33/50 [00:08<00:04, 3.85it/s]20)
68%|██████▊ | 34/50 [00:08<00:04, 3.85it/s]20)
70%|███████ | 35/50 [00:09<00:03, 3.84it/s]20)
72%|███████▏ | 36/50 [00:09<00:03, 3.84it/s]20)
74%|███████▍ | 37/50 [00:09<00:03, 3.84it/s]20)
76%|███████▌ | 38/50 [00:09<00:03, 3.84it/s]20)
78%|███████▊ | 39/50 [00:10<00:02, 3.84it/s]20)
80%|████████ | 40/50 [00:10<00:02, 3.84it/s]20)
82%|████████▏ | 41/50 [00:10<00:02, 3.84it/s]20)
84%|████████▍ | 42/50 [00:10<00:02, 3.84it/s]20)
86%|████████▌ | 43/50 [00:11<00:01, 3.84it/s]20)
88%|████████▊ | 44/50 [00:11<00:01, 3.84it/s]20)
90%|█████████ | 45/50 [00:11<00:01, 3.84it/s]20)
92%|█████████▏| 46/50 [00:11<00:01, 3.85it/s]20)
94%|█████████▍| 47/50 [00:12<00:00, 3.85it/s]20)
96%|█████████▌| 48/50 [00:12<00:00, 3.84it/s]20)
98%|█████████▊| 49/50 [00:12<00:00, 3.84it/s]20)
100%|██████████| 50/50 [00:13<00:00, 3.83it/s]20)
(ServeReplica:default:StableDiffusionV2 pid=25320) INFO 2024-02-07 17:58:36,604 default_StableDiffusionV2 OXPzZm 33133be7-246f-4492-9ab6-6a4c2666b306 /imagine replica.py:772 - GENERATE OK 14167.2ms
```
The app saves the `output.png` file locally. The following is an example of an output image. ![image](https://raw.githubusercontent.com/ray-project/images/master/docs/serve/stable_diffusion_inferentia2_output.png)
doc/source/serve/tutorials/aws-neuron-core-inference.md
+79
View File
@@ -0,0 +1,79 @@
---
orphan: true
---
(aws-neuron-core-inference-tutorial)=
# Serve an Inference Model on AWS NeuronCores Using FastAPI (Experimental)
This example compiles a BERT-based model and deploys the traced model on an AWS Inferentia (Inf2) or Tranium (Trn1) instance using Ray Serve and FastAPI.
:::{note}
Before starting this example:
* Set up [PyTorch Neuron](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/general/setup/torch-neuronx.html#setup-torch-neuronx)
* Install AWS NeuronCore drivers and tools, and torch-neuronx based on the instance-type
:::
```bash
python -m pip install "ray[serve]" requests transformers
```
This example uses the [j-hartmann/emotion-english-distilroberta-base](https://huggingface.co/j-hartmann/emotion-english-distilroberta-base) model and [FastAPI](https://fastapi.tiangolo.com/).
Use the following code to compile the model:
```{literalinclude} ../doc_code/aws_neuron_core_inference_serve.py
:language: python
:start-after: __compile_neuron_code_start__
:end-before: __compile_neuron_code_end__
```
For compiling the model, you should see the following log messages:
```text
Downloading (…)lve/main/config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.00k/1.00k [00:00<00:00, 242kB/s]
Downloading pytorch_model.bin: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 329M/329M [00:01<00:00, 217MB/s]
Downloading (…)okenizer_config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 294/294 [00:00<00:00, 305kB/s]
Downloading (…)olve/main/vocab.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 798k/798k [00:00<00:00, 22.0MB/s]
Downloading (…)olve/main/merges.txt: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 456k/456k [00:00<00:00, 57.0MB/s]
Downloading (…)/main/tokenizer.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.36M/1.36M [00:00<00:00, 6.16MB/s]
Downloading (…)cial_tokens_map.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 239/239 [00:00<00:00, 448kB/s]
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
- Avoid using `tokenizers` before the fork if possible
- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
Saved Neuron-compiled model ./sentiment_neuron.pt
```
The traced model should be ready for deployment. Save the following code to a file named aws_neuron_core_inference_serve.py.
Use `serve run aws_neuron_core_inference_serve:entrypoint` to start the Serve application.
```{literalinclude} ../doc_code/aws_neuron_core_inference_serve.py
:language: python
:start-after: __neuron_serve_code_start__
:end-before: __neuron_serve_code_end__
```
You should see the following log messages when a deployment is successful:
```text
(ServeController pid=43105) INFO 2023-08-23 20:29:32,694 controller 43105 deployment_state.py:1372 - Deploying new version of deployment default_BertBaseModel.
(ServeController pid=43105) INFO 2023-08-23 20:29:32,695 controller 43105 deployment_state.py:1372 - Deploying new version of deployment default_APIIngress.
(ProxyActor pid=43147) INFO 2023-08-23 20:29:32,620 http_proxy 10.0.1.234 http_proxy.py:1328 - Proxy actor 8be14f6b6b10c0190cd0c39101000000 starting on node 46a7f740898fef723c3360ef598c1309701b07d11fb9dc45e236620a.
(ProxyActor pid=43147) INFO: Started server process [43147]
(ServeController pid=43105) INFO 2023-08-23 20:29:32,799 controller 43105 deployment_state.py:1654 - Adding 1 replica to deployment default_BertBaseModel.
(ServeController pid=43105) INFO 2023-08-23 20:29:32,801 controller 43105 deployment_state.py:1654 - Adding 1 replica to deployment default_APIIngress.
2023-08-23 20:29:44,690 SUCC scripts.py:462 -- Deployed Serve app successfully.
```
Use the following code to send requests:
```python
import requests
response = requests.get(f"http://127.0.0.1:8000/infer?sentence=Ray is super cool")
print(response.status_code, response.json())
```
The response includes a status code and the classifier output:
```text
200 joy
```
doc/source/serve/tutorials/batch.md
+166
View File
@@ -0,0 +1,166 @@
---
orphan: true
---
(serve-batch-tutorial)=
# Serve a Text Generator with Request Batching
This tutorial shows how to deploy a text generator that processes multiple queries simultaneously using batching. Learn how to:
- Implement a Ray Serve deployment that handles batched requests
- Configure and optimize batch processing
- Query the model from HTTP and Python
Batching can significantly improve performance when your model supports parallel processing like GPU acceleration or vectorized operations. It increases both throughput and hardware utilization by processing multiple requests together.
:::{note}
This tutorial focuses on online serving with batching. For offline batch processing of large datasets, see [batch inference with Ray Data](batch_inference_home).
:::
## Prerequisites
```python
pip install "ray[serve] transformers"
```
## Define the Deployment
Open a new Python file called `tutorial_batch.py`. First, import Ray Serve and some other helpers.
```{literalinclude} ../doc_code/tutorial_batch.py
:end-before: __doc_import_end__
:start-after: __doc_import_begin__
```
Ray Serve provides the `@serve.batch` decorator to automatically batch individual requests to a function or class method.
The decorated method:
- Must be `async def` to handle concurrent requests
- Receives a list of requests to process together
- Returns a list of results of equal length, one for each request
```python
@serve.batch
async def my_batch_handler(self, requests: List):
# Process multiple requests together
results = []
for request in requests:
results.append(request) # processing logic here
return results
```
You can call the batch handler from another `async def` method in your deployment. Ray Serve batches and executes these calls together, but returns individual results just like normal function calls:
```python
class BatchingDeployment:
@serve.batch
async def my_batch_handler(self, requests: List):
results = []
for request in requests:
results.append(request.json()) # processing logic here
return results
async def __call__(self, request):
return await self.my_batch_handler(request)
```
:::{note}
Ray Serve uses *opportunistic batching* by default - executing requests as soon as they arrive without waiting for a full batch. You can adjust this behavior using `batch_wait_timeout_s` in the `@serve.batch` decorator to trade increased latency for increased throughput (defaults to 0). Increasing this value may improve throughput at the cost of latency under low load.
:::
Next, define a deployment that takes in a list of input strings and runs vectorized text generation on the inputs.
```{literalinclude} ../doc_code/tutorial_batch.py
:end-before: __doc_define_servable_end__
:start-after: __doc_define_servable_begin__
```
Next, prepare to deploy the deployment. Note that in the `@serve.batch` decorator, you are specifying the maximum batch size with `max_batch_size=4`. This option limits the maximum possible batch size that Ray Serve executes at once.
```{literalinclude} ../doc_code/tutorial_batch.py
:end-before: __doc_deploy_end__
:start-after: __doc_deploy_begin__
```
## Deployment Options
You can deploy your app in two ways:
### Option 1: Deploying with the Serve Command-Line Interface
```console
$ serve run tutorial_batch:generator --name "Text-Completion-App"
```
### Option 2: Deploying with the Python API
Alternatively, you can deploy the app using the Python API using the `serve.run` function. This command returns a handle that you can use to query the deployment.
```python
from ray.serve.handle import DeploymentHandle
handle: DeploymentHandle = serve.run(generator, name="Text-Completion-App")
```
You can now use this handle to query the model. See the [Querying the Model](#querying-the-model) section below.
## Querying the Model
There are multiple ways to interact with your deployed model:
### 1. Simple HTTP Queries
For basic testing, use curl:
```console
$ curl "http://localhost:8000/?text=Once+upon+a+time"
```
### 2. Send HTTP requests in parallel with Ray
For higher throughput, use [Ray remote tasks](ray-remote-functions) to send parallel requests:
```python
import ray
import requests
@ray.remote
def send_query(text):
resp = requests.post("http://localhost:8000/", params={"text": text})
return resp.text
# Example batch of queries
texts = [
'Once upon a time,',
'Hi my name is Lewis and I like to',
'In a galaxy far far away',
]
# Send all queries in parallel
results = ray.get([send_query.remote(text) for text in texts])
```
### 3. Sending requests using DeploymentHandle
For a more Pythonic way to query the model, you can use the deployment handle directly:
```python
import ray
from ray import serve
input_batch = [
'Once upon a time,',
'Hi my name is Lewis and I like to',
'In a galaxy far far away',
]
# initialize using the 'auto' option to connect to the already-running Ray cluster
ray.init(address="auto")
handle = serve.get_deployment_handle("BatchTextGenerator", app_name="Text-Completion-App")
responses = [handle.handle_batch.remote(text) for text in input_batch]
results = [r.result() for r in responses]
```
## Performance Considerations
- Increase `max_batch_size` if you have sufficient memory and want higher throughput - this may increase latency
- Increase `batch_wait_timeout_s` if throughput is more important than latency
- Increase `max_concurrent_batches` if you have an asynchronous function that you want to process multiple batches with concurrently
doc/source/serve/tutorials/gradio-integration.md
+143
View File
@@ -0,0 +1,143 @@
---
orphan: true
---
# Scale a Gradio App with Ray Serve
This guide shows how to scale up your [Gradio](https://gradio.app/) application using Ray Serve. Keep the internal architecture of your Gradio app intact, with no code changes. Simply wrap the app within Ray Serve as a deployment and scale it to access more resources.
This tutorial uses Gradio apps that run text summarization and generation models and use [Hugging Face's Pipelines](https://huggingface.co/docs/transformers/main_classes/pipelines) to access these models.
:::{note}
Remember you can substitute this app with your own Gradio app if you want to try scaling up your own Gradio app.
:::
## Dependencies
To follow this tutorial, you need Ray Serve, Gradio, and transformers. If you haven't already, install them by running:
```console
$ pip install "ray[serve]" gradio==3.50.2 torch transformers
```
## Example 1: Scaling up your Gradio app with `GradioServer`
The first example summarizes text using the [T5 Small](https://huggingface.co/t5-small) model and uses [Hugging Face's Pipelines](https://huggingface.co/docs/transformers/main_classes/pipelines) to access that model. It demonstrates one easy way to deploy Gradio apps onto Ray Serve: using the simple `GradioServer` wrapper. Example 2 shows how to use `GradioIngress` for more customized use-cases.
First, create a new Python file named `demo.py`. Second, import `GradioServer` from Ray Serve to deploy your Gradio app later, `gradio`, and `transformers.pipeline` to load text summarization models.
```{literalinclude} ../doc_code/gradio-integration.py
:start-after: __doc_import_begin__
:end-before: __doc_import_end__
```
Then, write a builder function that constructs the Gradio app `io`.
```{literalinclude} ../doc_code/gradio-integration.py
:start-after: __doc_gradio_app_begin__
:end-before: __doc_gradio_app_end__
```
### Deploying Gradio Server
To deploy your Gradio app onto Ray Serve, you need to wrap the Gradio app in a Serve [deployment](serve-key-concepts-deployment). `GradioServer` acts as that wrapper. It serves your Gradio app remotely on Ray Serve so that it can process and respond to HTTP requests.
By wrapping your application in `GradioServer`, you can increase the number of CPUs and/or GPUs available to the application.
:::{note}
Ray Serve doesn't support routing requests to multiple replicas of `GradioServer`, so you should only have a single replica.
:::
:::{note}
`GradioServer` is simply `GradioIngress` but wrapped in a Serve deployment. You can use `GradioServer` for the simple wrap-and-deploy use case, but in the next section, you can use `GradioIngress` to define your own Gradio Server for more customized use cases.
:::
:::{note}
Ray can’t pickle Gradio. Instead, pass a builder function that constructs the Gradio interface.
:::
Using either the Gradio app `io`, which the builder function constructed, or your own Gradio app of type `Interface`, `Block`, `Parallel`, etc., wrap the app in your Gradio Server. Pass the builder function as input to your Gradio Server. Ray Serves uses the builder function to construct your Gradio app on the Ray cluster.
```{literalinclude} ../doc_code/gradio-integration.py
:start-after: __doc_app_begin__
:end-before: __doc_app_end__
```
Finally, deploy your Gradio Server. Run the following in your terminal, assuming that you saved the file as `demo.py`:
```console
$ serve run demo:app
```
Access your Gradio app at `http://localhost:8000` The output should look like the following image: ![Gradio Result](https://raw.githubusercontent.com/ray-project/images/master/docs/serve/gradio_result.png)
See the [Production Guide](serve-in-production) for more information on how to deploy your app in production.
## Example 2: Parallelizing models with Ray Serve
You can run multiple models in parallel with Ray Serve by using [model composition](serve-model-composition) in Ray Serve.
Suppose you want to run the following program.
1. Take two text generation models, [`gpt2`](https://huggingface.co/gpt2) and [`distilgpt2`](https://huggingface.co/distilgpt2).
2. Run the two models on the same input text, so that the generated text has a minimum length of 20 and maximum length of 100.
3. Display the outputs of both models using Gradio.
The following is a comparison of an unparallelized approach using vanilla Gradio to a parallelized approach using Ray Serve.
### Vanilla Gradio
This code is a typical implementation:
```{literalinclude} ../doc_code/gradio-original.py
:start-after: __doc_code_begin__
:end-before: __doc_code_end__
```
Launch the Gradio app with this command:
```
demo.launch()
```
### Parallelize using Ray Serve
With Ray Serve, you can parallelize the two text generation models by wrapping each model in a separate Ray Serve [deployment](serve-key-concepts-deployment). You can define deployments by decorating a Python class or function with `@serve.deployment`. The deployments usually wrap the models that you want to deploy on Ray Serve to handle incoming requests.
Follow these steps to achieve parallelism. First, import the dependencies. Note that you need to import `GradioIngress` instead of `GradioServer` like before because in this case, you're building a customized `MyGradioServer` that can run models in parallel.
```{literalinclude} ../doc_code/gradio-integration-parallel.py
:start-after: __doc_import_begin__
:end-before: __doc_import_end__
```
Then, wrap the `gpt2` and `distilgpt2` models in Serve deployments, named `TextGenerationModel`.
```{literalinclude} ../doc_code/gradio-integration-parallel.py
:start-after: __doc_models_begin__
:end-before: __doc_models_end__
```
Next, instead of simply wrapping the Gradio app in a `GradioServer` deployment, build your own `MyGradioServer` that reroutes the Gradio app so that it runs the `TextGenerationModel` deployments.
```{literalinclude} ../doc_code/gradio-integration-parallel.py
:start-after: __doc_gradio_server_begin__
:end-before: __doc_gradio_server_end__
```
Lastly, link everything together:
```{literalinclude} ../doc_code/gradio-integration-parallel.py
:start-after: __doc_app_begin__
:end-before: __doc_app_end__
```
:::{note}
This step binds the two text generation models, which you wrapped in Serve deployments, to `MyGradioServer._d1` and `MyGradioServer._d2`, forming a [model composition](serve-model-composition). In the example, the Gradio Interface `io` calls `MyGradioServer.fanout()`, which sends requests to the two text generation models that you deployed on Ray Serve.
:::
Now, you can run your scalable app, to serve the two text generation models in parallel on Ray Serve. Run your Gradio app with the following command:
```console
$ serve run demo:app
```
Access your Gradio app at `http://localhost:8000`, and you should see the following interactive interface: ![Gradio Result](https://raw.githubusercontent.com/ray-project/images/master/docs/serve/gradio_result_parallel.png)
See the [Production Guide](serve-in-production) for more information on how to deploy your app in production.
doc/source/serve/tutorials/intel-gaudi-inference.md
+287
View File
@@ -0,0 +1,287 @@
---
orphan: true
---
# Serve Llama2-7b/70b on a single or multiple Intel Gaudi Accelerator
[Intel Gaudi AI Processors (HPUs)](https://habana.ai) are AI hardware accelerators designed by Intel Habana Labs. See [Gaudi Architecture](https://docs.habana.ai/en/latest/Gaudi_Overview/index.html) and [Gaudi Developer Docs](https://developer.habana.ai/) for more details.
This tutorial has two examples:
1. Deployment of [Llama2-7b](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf) using a single HPU:
* Load a model onto an HPU.
* Perform generation on an HPU.
* Enable HPU Graph optimizations.
2. Deployment of [Llama2-70b](https://huggingface.co/meta-llama/Llama-2-70b-chat-hf) using multiple HPUs on a single node:
* Initialize a distributed backend.
* Load a sharded model onto DeepSpeed workers.
* Stream responses from DeepSpeed workers.
This tutorial serves a large language model (LLM) on HPUs.
## Environment setup
Use a prebuilt container to run these examples. To run a container, you need Docker. See [Install Docker Engine](https://docs.docker.com/engine/install/) for installation instructions.
Next, follow [Run Using Containers](https://docs.habana.ai/en/latest/Installation_Guide/Bare_Metal_Fresh_OS.html?highlight=installer#run-using-containers) to install the Gaudi drivers and container runtime. To verify your installation, start a shell and run `hl-smi`. It should print status information about the HPUs on the machine:
```text
+-----------------------------------------------------------------------------+
| HL-SMI Version: hl-1.20.0-fw-58.1.1.1 |
| Driver Version: 1.19.1-6f47ddd |
| Nic Driver Version: 1.19.1-f071c23 |
|-------------------------------+----------------------+----------------------+
| AIP Name Persistence-M| Bus-Id Disp.A | Volatile Uncor-Events|
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | AIP-Util Compute M. |
|===============================+======================+======================|
| 0 HL-225 N/A | 0000:9a:00.0 N/A | 0 |
| N/A 22C N/A 96W / 600W | 768MiB / 98304MiB | 0% 0% |
|-------------------------------+----------------------+----------------------+
| 1 HL-225 N/A | 0000:9b:00.0 N/A | 0 |
| N/A 24C N/A 78W / 600W | 768MiB / 98304MiB | 0% 0% |
|-------------------------------+----------------------+----------------------+
| 2 HL-225 N/A | 0000:b3:00.0 N/A | 0 |
| N/A 25C N/A 81W / 600W | 768MiB / 98304MiB | 0% 0% |
|-------------------------------+----------------------+----------------------+
| 3 HL-225 N/A | 0000:b4:00.0 N/A | 0 |
| N/A 22C N/A 92W / 600W | 96565MiB / 98304MiB | 0% 98% |
|-------------------------------+----------------------+----------------------+
| 4 HL-225 N/A | 0000:33:00.0 N/A | 0 |
| N/A 22C N/A 83W / 600W | 768MiB / 98304MiB | 0% 0% |
|-------------------------------+----------------------+----------------------+
| 5 HL-225 N/A | 0000:4e:00.0 N/A | 0 |
| N/A 21C N/A 80W / 600W | 96564MiB / 98304MiB | 0% 98% |
|-------------------------------+----------------------+----------------------+
| 6 HL-225 N/A | 0000:34:00.0 N/A | 0 |
| N/A 25C N/A 86W / 600W | 768MiB / 98304MiB | 0% 0% |
|-------------------------------+----------------------+----------------------+
| 7 HL-225 N/A | 0000:4d:00.0 N/A | 0 |
| N/A 30C N/A 100W / 600W | 17538MiB / 98304MiB | 0% 17% |
|-------------------------------+----------------------+----------------------+
| Compute Processes: AIP Memory |
| AIP PID Type Process name Usage |
|=============================================================================|
| 0 N/A N/A N/A N/A |
| 1 N/A N/A N/A N/A |
| 2 N/A N/A N/A N/A |
| 3 N/A N/A N/A N/A |
| 4 N/A N/A N/A N/A |
| 5 N/A N/A N/A N/A |
| 6 N/A N/A N/A N/A |
| 7 107684 C ray::_RayTrainW 16770MiB
+=============================================================================+
```
Next, start the Gaudi container:
```bash
docker pull vault.habana.ai/gaudi-docker/1.20.0/ubuntu22.04/habanalabs/pytorch-installer-2.6.0:latest
docker run -it --runtime=habana -e HABANA_VISIBLE_DEVICES=all -e OMPI_MCA_btl_vader_single_copy_mechanism=none --cap-add=sys_nice --net=host --ipc=host vault.habana.ai/gaudi-docker/1.20.0/ubuntu22.04/habanalabs/pytorch-installer-2.6.0:latest
```
To follow the examples in this tutorial, mount the directory containing the examples and models into the container. Inside the container, run:
```bash
pip install ray[tune,serve]
pip install git+https://github.com/huggingface/optimum-habana.git
# Replace 1.20.0 with the driver version of the container.
pip install git+https://github.com/HabanaAI/DeepSpeed.git@1.20.0
# Only needed by the DeepSpeed example.
export RAY_EXPERIMENTAL_NOSET_HABANA_VISIBLE_MODULES=1
```
Start Ray in the container with `ray start --head`. You are now ready to run the examples.
## Running a model on a single HPU
This example shows how to deploy a Llama2-7b model on an HPU for inference.
First, define a deployment that serves a Llama2-7b model using an HPU. Note that we enable [HPU graph optimizations](https://docs.habana.ai/en/latest/Gaudi_Overview/SynapseAI_Software_Suite.html?highlight=graph#graph-compiler-and-runtime) for better performance.
```{literalinclude} ../doc_code/intel_gaudi_inference_serve.py
:language: python
:start-after: __model_def_start__
:end-before: __model_def_end__
```
Copy the code above and save it as `intel_gaudi_inference_serve.py`. Start the deployment like this:
```bash
serve run intel_gaudi_inference_serve:entrypoint
```
The terminal should print logs as the deployment starts up:
```text
2025-03-03 06:07:08,106 INFO scripts.py:494 -- Running import path: 'infer:entrypoint'.
2025-03-03 06:07:09,295 INFO worker.py:1654 -- Connecting to existing Ray cluster at address: 100.83.111.228:6379...
2025-03-03 06:07:09,304 INFO worker.py:1832 -- Connected to Ray cluster. View the dashboard at 127.0.0.1:8265
(ProxyActor pid=147082) INFO 2025-03-03 06:07:11,096 proxy 100.83.111.228 -- Proxy starting on node b4d028b67678bfdd190b503b44780bc319c07b1df13ac5c577873861 (HTTP port: 8000).
INFO 2025-03-03 06:07:11,202 serve 162730 -- Started Serve in namespace "serve".
INFO 2025-03-03 06:07:11,203 serve 162730 -- Connecting to existing Serve app in namespace "serve". New http options will not be applied.
(ProxyActor pid=147082) INFO 2025-03-03 06:07:11,184 proxy 100.83.111.228 -- Got updated endpoints: {}.
(ServeController pid=147087) INFO 2025-03-03 06:07:11,278 controller 147087 -- Deploying new version of Deployment(name='LlamaModel', app='default') (initial target replicas: 1).
(ProxyActor pid=147082) INFO 2025-03-03 06:07:11,280 proxy 100.83.111.228 -- Got updated endpoints: {Deployment(name='LlamaModel', app='default'): EndpointInfo(route='/', app_is_cross_language=False)}.
(ProxyActor pid=147082) INFO 2025-03-03 06:07:11,286 proxy 100.83.111.228 -- Started <ray.serve._private.router.SharedRouterLongPollClient object at 0x7f74804e90c0>.
(ServeController pid=147087) INFO 2025-03-03 06:07:11,381 controller 147087 -- Adding 1 replica to Deployment(name='LlamaModel', app='default').
(ServeReplica:default:LlamaModel pid=147085) [WARNING|utils.py:212] 2025-03-03 06:07:15,251 >> optimum-habana v1.15.0 has been validated for SynapseAI v1.19.0 but habana-frameworks v1.20.0.543 was found, this could lead to undefined behavior!
(ServeReplica:default:LlamaModel pid=147085) /usr/local/lib/python3.10/dist-packages/transformers/deepspeed.py:24: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations
(ServeReplica:default:LlamaModel pid=147085) warnings.warn(
(ServeReplica:default:LlamaModel pid=147085) /usr/local/lib/python3.10/dist-packages/transformers/models/auto/tokenization_auto.py:796: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
(ServeReplica:default:LlamaModel pid=147085) warnings.warn(
(ServeReplica:default:LlamaModel pid=147085) /usr/local/lib/python3.10/dist-packages/transformers/models/auto/configuration_auto.py:991: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
(ServeReplica:default:LlamaModel pid=147085) warnings.warn(
(ServeReplica:default:LlamaModel pid=147085) /usr/local/lib/python3.10/dist-packages/transformers/models/auto/auto_factory.py:471: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
(ServeReplica:default:LlamaModel pid=147085) warnings.warn(
Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards: 50%|█████ | 1/2 [00:01<00:01, 1.72s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:02<00:00, 1.45s/it]
(ServeReplica:default:LlamaModel pid=147085) ============================= HABANA PT BRIDGE CONFIGURATION ===========================
(ServeReplica:default:LlamaModel pid=147085) PT_HPU_LAZY_MODE = 1
(ServeReplica:default:LlamaModel pid=147085) PT_HPU_RECIPE_CACHE_CONFIG = ,false,1024
(ServeReplica:default:LlamaModel pid=147085) PT_HPU_MAX_COMPOUND_OP_SIZE = 9223372036854775807
(ServeReplica:default:LlamaModel pid=147085) PT_HPU_LAZY_ACC_PAR_MODE = 1
(ServeReplica:default:LlamaModel pid=147085) PT_HPU_ENABLE_REFINE_DYNAMIC_SHAPES = 0
(ServeReplica:default:LlamaModel pid=147085) PT_HPU_EAGER_PIPELINE_ENABLE = 1
(ServeReplica:default:LlamaModel pid=147085) PT_HPU_EAGER_COLLECTIVE_PIPELINE_ENABLE = 1
(ServeReplica:default:LlamaModel pid=147085) PT_HPU_ENABLE_LAZY_COLLECTIVES = 0
(ServeReplica:default:LlamaModel pid=147085) ---------------------------: System Configuration :---------------------------
(ServeReplica:default:LlamaModel pid=147085) Num CPU Cores : 160
(ServeReplica:default:LlamaModel pid=147085) CPU RAM : 1056374420 KB
(ServeReplica:default:LlamaModel pid=147085) ------------------------------------------------------------------------------
INFO 2025-03-03 06:07:30,359 serve 162730 -- Application 'default' is ready at http://127.0.0.1:8000/.
INFO 2025-03-03 06:07:30,359 serve 162730 -- Deployed app 'default' successfully.
```
In another shell, use the following code to send requests to the deployment to perform generation tasks.
```{literalinclude} ../doc_code/intel_gaudi_inference_client.py
:language: python
:start-after: __main_code_start__
:end-before: __main_code_end__
```
Here is an example output:
```text
Once upon a time, in a small village nestled in the rolling hills of Tuscany, there lived a young girl named Sophia.
Sophia was a curious and adventurous child, always eager to explore the world around her. She spent her days playing in the fields and forests, chasing after butterflies and watching the clouds drift lazily across the sky.
One day, as Sophia was wandering through the village, she stumbled upon a beautiful old book hidden away in a dusty corner of the local library. The book was bound in worn leather and adorned with intr
in a small village nestled in the rolling hills of Tuscany, there lived a young girl named Luna.
Luna was a curious and adventurous child, always eager to explore the world around her. She spent her days wandering through the village, discovering new sights and sounds at every turn.
One day, as she was wandering through the village, Luna stumbled upon a hidden path she had never seen before. The path was overgrown with weeds and vines, and it seemed to disappear into the distance.
Luna's curiosity was piqued,
```
## Running a sharded model on multiple HPUs
This example deploys a Llama2-70b model using 8 HPUs orchestrated by DeepSpeed.
The example requires caching the Llama2-70b model. Run the following Python code in the Gaudi container to cache the model.
```python
from huggingface_hub import snapshot_download
snapshot_download(
"meta-llama/Llama-2-70b-chat-hf",
# Replace the path if necessary.
cache_dir=os.getenv("TRANSFORMERS_CACHE", None),
# Specify your Hugging Face token.
token=""
)
```
In this example, the deployment replica sends prompts to the DeepSpeed workers, which are running in Ray actors:
```{literalinclude} ../doc_code/intel_gaudi_inference_serve_deepspeed.py
:language: python
:start-after: __worker_def_start__
:end-before: __worker_def_end__
```
Next, define a deployment:
```{literalinclude} ../doc_code/intel_gaudi_inference_serve_deepspeed.py
:language: python
:start-after: __deploy_def_start__
:end-before: __deploy_def_end__
```
Copy both blocks of the preceding code and save them into `intel_gaudi_inference_serve_deepspeed.py`. Run this example using `serve run intel_gaudi_inference_serve_deepspeed:entrypoint`.
Notice!!! Please set the environment variable `HABANA_VISIBLE_MODULES` carefully.
The terminal should print logs as the deployment starts up:
```text
2025-03-03 06:21:57,692 INFO scripts.py:494 -- Running import path: 'infer-ds:entrypoint'.
2025-03-03 06:22:03,064 INFO worker.py:1832 -- Started a local Ray instance. View the dashboard at 127.0.0.1:8265
INFO 2025-03-03 06:22:07,343 serve 170212 -- Started Serve in namespace "serve".
INFO 2025-03-03 06:22:07,343 serve 170212 -- Connecting to existing Serve app in namespace "serve". New http options will not be applied.
(ServeController pid=170719) INFO 2025-03-03 06:22:07,377 controller 170719 -- Deploying new version of Deployment(name='DeepSpeedLlamaModel', app='default') (initial target replicas: 1).
(ProxyActor pid=170723) INFO 2025-03-03 06:22:07,290 proxy 100.83.111.228 -- Proxy starting on node 47721c925467a877497e66104328bb72dc7bd7f900a63b2f1fdb48b2 (HTTP port: 8000).
(ProxyActor pid=170723) INFO 2025-03-03 06:22:07,325 proxy 100.83.111.228 -- Got updated endpoints: {}.
(ProxyActor pid=170723) INFO 2025-03-03 06:22:07,379 proxy 100.83.111.228 -- Got updated endpoints: {Deployment(name='DeepSpeedLlamaModel', app='default'): EndpointInfo(route='/', app_is_cross_language=False)}.
(ServeController pid=170719) INFO 2025-03-03 06:22:07,478 controller 170719 -- Adding 1 replica to Deployment(name='DeepSpeedLlamaModel', app='default').
(ProxyActor pid=170723) INFO 2025-03-03 06:22:07,422 proxy 100.83.111.228 -- Started <ray.serve._private.router.SharedRouterLongPollClient object at 0x7fa557945210>.
(DeepSpeedInferenceWorker pid=179962) [WARNING|utils.py:212] 2025-03-03 06:22:14,611 >> optimum-habana v1.15.0 has been validated for SynapseAI v1.19.0 but habana-frameworks v1.20.0.543 was found, this could lead to undefined behavior!
(DeepSpeedInferenceWorker pid=179963) /usr/local/lib/python3.10/dist-packages/transformers/deepspeed.py:24: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations
(DeepSpeedInferenceWorker pid=179963) warnings.warn(
(DeepSpeedInferenceWorker pid=179964) [WARNING|utils.py:212] 2025-03-03 06:22:14,613 >> optimum-habana v1.15.0 has been validated for SynapseAI v1.19.0 but habana-frameworks v1.20.0.543 was found, this could lead to undefined behavior! [repeated 3x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)
(DeepSpeedInferenceWorker pid=179962) [2025-03-03 06:22:23,502] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to hpu (auto detect)
Loading 2 checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]
(DeepSpeedInferenceWorker pid=179962) [2025-03-03 06:22:24,032] [INFO] [logging.py:105:log_dist] [Rank -1] DeepSpeed info: version=0.16.1+hpu.synapse.v1.20.0, git-hash=61543a96, git-branch=1.20.0
(DeepSpeedInferenceWorker pid=179962) [2025-03-03 06:22:24,035] [INFO] [logging.py:105:log_dist] [Rank -1] quantize_bits = 8 mlp_extra_grouping = False, quantize_groups = 1
(DeepSpeedInferenceWorker pid=179962) [2025-03-03 06:22:24,048] [INFO] [comm.py:652:init_distributed] cdb=None
(DeepSpeedInferenceWorker pid=179963) ============================= HABANA PT BRIDGE CONFIGURATION ===========================
(DeepSpeedInferenceWorker pid=179963) PT_HPU_LAZY_MODE = 1
(DeepSpeedInferenceWorker pid=179963) PT_HPU_RECIPE_CACHE_CONFIG = ,false,1024
(DeepSpeedInferenceWorker pid=179963) PT_HPU_MAX_COMPOUND_OP_SIZE = 9223372036854775807
(DeepSpeedInferenceWorker pid=179963) PT_HPU_LAZY_ACC_PAR_MODE = 0
(DeepSpeedInferenceWorker pid=179963) PT_HPU_ENABLE_REFINE_DYNAMIC_SHAPES = 0
(DeepSpeedInferenceWorker pid=179963) PT_HPU_EAGER_PIPELINE_ENABLE = 1
(DeepSpeedInferenceWorker pid=179963) PT_HPU_EAGER_COLLECTIVE_PIPELINE_ENABLE = 1
(DeepSpeedInferenceWorker pid=179963) PT_HPU_ENABLE_LAZY_COLLECTIVES = 1
(DeepSpeedInferenceWorker pid=179963) ---------------------------: System Configuration :---------------------------
(DeepSpeedInferenceWorker pid=179963) Num CPU Cores : 160
(DeepSpeedInferenceWorker pid=179963) CPU RAM : 1056374420 KB
(DeepSpeedInferenceWorker pid=179963) ------------------------------------------------------------------------------
(DeepSpeedInferenceWorker pid=179964) /usr/local/lib/python3.10/dist-packages/transformers/deepspeed.py:24: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations [repeated 3x across cluster]
(DeepSpeedInferenceWorker pid=179964) warnings.warn( [repeated 3x across cluster]
Loading 2 checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s] [repeated 3x across cluster]
(ServeController pid=170719) WARNING 2025-03-03 06:22:37,562 controller 170719 -- Deployment 'DeepSpeedLlamaModel' in application 'default' has 1 replicas that have taken more than 30s to initialize.
(ServeController pid=170719) This may be caused by a slow __init__ or reconfigure method.
Loading 2 checkpoint shards: 50%|█████ | 1/2 [00:17<00:17, 17.51s/it]
Loading 2 checkpoint shards: 100%|██████████| 2/2 [00:21<00:00, 9.57s/it]
Loading 2 checkpoint shards: 100%|██████████| 2/2 [00:21<00:00, 10.88s/it]
Loading 2 checkpoint shards: 50%|█████ | 1/2 [00:18<00:18, 18.70s/it] [repeated 3x across cluster]
INFO 2025-03-03 06:22:48,569 serve 170212 -- Application 'default' is ready at http://127.0.0.1:8000/.
INFO 2025-03-03 06:22:48,569 serve 170212 -- Deployed app 'default' successfully.
```
Use the same code snippet introduced in the single HPU example to send generation requests. Here's an example output:
```text
Once upon a time, in a far-off land, there was a magical kingdom called "Happily Ever Laughter." It was a place where laughter was the key to unlocking all the joys of life, and where everyone lived in perfect harmony.
In this kingdom, there was a beautiful princess named Lily. She was kind, gentle, and had a heart full of laughter. Every day, she would wake up with a big smile on her face, ready to face whatever adventures the day might bring.
One day, a wicked sorcerer cast a spell on the kingdom
Once upon a time, in a far-off land, there was a magical kingdom called "Happily Ever Laughter." It was a place where laughter was the key to unlocking all the joys of life, and where everyone lived in perfect harmony.
In this kingdom, there was a beautiful princess named Lily. She was kind, gentle, and had a heart full of laughter. Every day, she would wake up with a big smile on her face, ready to face whatever adventures the day might bring.
One day, a wicked sorcerer cast a spell on the kingdom
```
## Next Steps
See [llm-on-ray](https://github.com/intel/llm-on-ray) for more ways to customize and deploy LLMs at scale.
doc/source/serve/tutorials/java.md
+148
View File
@@ -0,0 +1,148 @@
---
orphan: true
---
(serve-java-tutorial)=
# Serve a Java App
To use Java Ray Serve, you need the following dependency in your pom.xml.
```xml
<dependency>
<groupId>io.ray</groupId>
<artifactId>ray-serve</artifactId>
<version>${ray.version}</version>
<scope>provided</scope>
</dependency>
```
> NOTE: After installing Ray with Python, the local environment includes the Java jar of Ray Serve. The `provided` scope ensures that you can compile the Java code using Ray Serve without version conflicts when you deploy on the cluster.
## Example model
This example use case is a production workflow for a financial application. The application needs to compute the best strategy to interact with different banks for a single task.
```{literalinclude} ../../../../java/serve/src/test/java/io/ray/serve/docdemo/Strategy.java
:end-before: docs-strategy-end
:language: java
:start-after: docs-strategy-start
```
This example uses the `Strategy` class to calculate the indicators of a number of banks.
* The `calc` method is the entry of the calculation. The input parameters are the time interval of calculation and the map of the banks and their indicators. The `calc` method contains a two-tier `for` loop, traversing each indicator list of each bank, and calling the `calcBankIndicators` method to calculate the indicators of the specified bank.
- There is another layer of `for` loop in the `calcBankIndicators` method, which traverses each indicator, and then calls the `calcIndicator` method to calculate the specific indicator of the bank.
- The `calcIndicator` method is a specific calculation logic based on the bank, the specified time interval and the indicator.
This code uses the `Strategy` class:
```{literalinclude} ../../../../java/serve/src/test/java/io/ray/serve/docdemo/StrategyCalc.java
:end-before: docs-strategy-calc-end
:language: java
:start-after: docs-strategy-calc-start
```
When the scale of banks and indicators expands, the three-tier `for` loop slows down the calculation. Even if you use the thread pool to calculate each indicator in parallel, you may encounter a single machine performance bottleneck. Moreover, you can't use this `Strategy` object as a resident service.
## Converting to a Ray Serve Deployment
Through Ray Serve, you can deploy the core computing logic of `Strategy` as a scalable distributed computing service.
First, extract the indicator calculation of each institution into a separate `StrategyOnRayServe` class:
```{literalinclude} ../../../../java/serve/src/test/java/io/ray/serve/docdemo/StrategyOnRayServe.java
:end-before: docs-strategy-end
:language: java
:start-after: docs-strategy-start
```
Next, start the Ray Serve runtime and deploy `StrategyOnRayServe` as a deployment.
```{literalinclude} ../../../../java/serve/src/test/java/io/ray/serve/docdemo/StrategyCalcOnRayServe.java
:end-before: docs-deploy-end
:language: java
:start-after: docs-deploy-start
```
The `Deployment.create` makes a Deployment object named `strategy`. After executing `Deployment.deploy`, the Ray Serve instance deploys this `strategy` deployment with four replicas, and you can access it for distributed parallel computing.
## Testing the Ray Serve Deployment
You can test the `strategy` deployment using RayServeHandle inside Ray:
```{literalinclude} ../../../../java/serve/src/test/java/io/ray/serve/docdemo/StrategyCalcOnRayServe.java
:end-before: docs-calc-end
:language: java
:start-after: docs-calc-start
```
This code executes the calculation of each bank's indicator serially, and sends it to Ray for execution. You can make the calculation concurrent, which not only improves the calculation efficiency, but also solves the bottleneck of single machine.
```{literalinclude} ../../../../java/serve/src/test/java/io/ray/serve/docdemo/StrategyCalcOnRayServe.java
:end-before: docs-parallel-calc-end
:language: java
:start-after: docs-parallel-calc-start
```
You can use `StrategyCalcOnRayServe` like the example in the `main` method:
```{literalinclude} ../../../../java/serve/src/test/java/io/ray/serve/docdemo/StrategyCalcOnRayServe.java
:end-before: docs-main-end
:language: java
:start-after: docs-main-start
```
## Calling Ray Serve Deployment with HTTP
Another way to test or call a deployment is through the HTTP request. However, two limitations exist for the Java deployments:
- Only the `call` method of the user class can process the HTTP requests.
- The `call` method can only have one input parameter, and the type of the input parameter and the returned value can only be `String`.
If you want to call the `strategy` deployment with HTTP, then you can rewrite the class like this code:
```{literalinclude} ../../../../java/serve/src/test/java/io/ray/serve/docdemo/HttpStrategyOnRayServe.java
:end-before: docs-strategy-end
:language: java
:start-after: docs-strategy-start
```
After deploying this deployment, you can access it with the `curl` command:
```shell
curl -d '{"time":1641038674, "bank":"test_bank", "indicator":"test_indicator"}' http://127.0.0.1:8000/strategy
```
You can also access it using HTTP Client in Java code:
```{literalinclude} ../../../../java/serve/src/test/java/io/ray/serve/docdemo/HttpStrategyCalcOnRayServe.java
:end-before: docs-http-end
:language: java
:start-after: docs-http-start
```
The example of strategy calculation using HTTP to access deployment is as follows:
```{literalinclude} ../../../../java/serve/src/test/java/io/ray/serve/docdemo/HttpStrategyCalcOnRayServe.java
:end-before: docs-calc-end
:language: java
:start-after: docs-calc-start
```
You can also rewrite this code to support concurrency:
```{literalinclude} ../../../../java/serve/src/test/java/io/ray/serve/docdemo/HttpStrategyCalcOnRayServe.java
:end-before: docs-parallel-calc-end
:language: java
:start-after: docs-parallel-calc-start
```
Finally, the complete usage of `HttpStrategyCalcOnRayServe` is like this code:
```{literalinclude} ../../../../java/serve/src/test/java/io/ray/serve/docdemo/HttpStrategyCalcOnRayServe.java
:end-before: docs-main-end
:language: java
:start-after: docs-main-start
```
doc/source/serve/tutorials/object-detection.md
+114
View File
@@ -0,0 +1,114 @@
---
orphan: true
---
(serve-object-detection-tutorial)=
# Building a Real-time Object Detection Service with Ray Serve
## Overview
This tutorial demonstrates how to deploy a production-ready object detection service using Ray Serve. You will learn how to serve a YOLOv5 object detection model efficiently with automatic GPU resource management and scaling capabilities.
## Installation
Install the required dependencies:
```bash
pip install "ray[serve]" requests torch pillow numpy opencv-python-headless pandas "gitpython>=3.1.30"
```
## Implementation
This example uses the [ultralytics/yolov5](https://github.com/ultralytics/yolov5) model for object detection and [FastAPI](https://fastapi.tiangolo.com/) for creating the web API.
### Code Structure
Save the following code to a file named `object_detection.py`:
```{literalinclude} ../doc_code/object_detection.py
:language: python
:start-after: __example_code_start__
:end-before: __example_code_end__
```
The code consists of two main deployments:
1. **APIIngress**: A FastAPI-based frontend that handles HTTP requests
2. **ObjectDetection**: The backend deployment that loads the YOLOv5 model and performs inference on GPU
:::{note}
**Understanding Autoscaling**
The configuration in this example sets `min_replicas` to 0, which means:
- The deployment starts with no `ObjectDetection` replicas
- Ray Serve creates replicas only when requests arrive
- After a period of inactivity, Ray Serve scales down the replicas back to 0
- This "scale-to-zero" capability helps conserve GPU resources when the service isn't being actively used
:::
## Deployment
Deploy the service with:
```bash
serve run object_detection:entrypoint
```
When successfully deployed, you should see log messages similar to:
```text
(ServeReplica:ObjectDection pid=4747) warnings.warn(
(ServeReplica:ObjectDection pid=4747) Downloading: "https://github.com/ultralytics/yolov5/zipball/master" to /home/ray/.cache/torch/hub/master.zip
(ServeReplica:ObjectDection pid=4747) YOLOv5 🚀 2023-3-8 Python-3.9.16 torch-1.13.0+cu116 CUDA:0 (Tesla T4, 15110MiB)
(ServeReplica:ObjectDection pid=4747)
(ServeReplica:ObjectDection pid=4747) Fusing layers...
(ServeReplica:ObjectDection pid=4747) YOLOv5s summary: 213 layers, 7225885 parameters, 0 gradients
(ServeReplica:ObjectDection pid=4747) Adding AutoShape...
2023-03-08 21:10:21,685 SUCC <string>:93 -- Deployed Serve app successfully.
```
## Troubleshooting
:::{tip}
**Common OpenCV Error**
You might encounter this error when running the example:
```
ImportError: libGL.so.1: cannot open shared object file: No such file or directory
```
This typically happens when running `opencv-python` in headless environments like containers. The solution is to use the headless version:
```bash
pip uninstall opencv-python; pip install opencv-python-headless
```
:::
## Testing the Service
Once the service is running, you can test it with the following Python code:
```python
import requests
# Sample image URL for testing
image_url = "https://ultralytics.com/images/zidane.jpg"
# Send request to the object detection service
resp = requests.get(f"http://127.0.0.1:8000/detect?image_url={image_url}")
# Save the annotated image with detected objects
with open("output.jpeg", 'wb') as f:
f.write(resp.content)
```
## Example Output
The service processes the image and returns it with bounding boxes around detected objects:
![Example of object detection output](https://raw.githubusercontent.com/ray-project/images/master/docs/serve/object_detection_output.jpeg)
doc/source/serve/tutorials/serve-deepseek.md
+170
View File
@@ -0,0 +1,170 @@
---
orphan: true
---
(serve-deepseek-tutorial)=
# Serve DeepSeek
This example shows how to deploy DeepSeek R1 or V3 with Ray Serve LLM.
## Installation
To run this example, install the following:
```bash
pip install "ray[llm]==2.46.0"
```
Note: Deploying DeepSeek-R1 requires at least 720GB of free disk space per worker node to store model weights.
## Deployment
### Quick Deployment
For quick deployment and testing, save the following code to a file named `deepseek.py`, and run `python3 deepseek.py`.
```python
from ray import serve
from ray.serve.llm import LLMConfig, build_openai_app
llm_config = LLMConfig(
model_loading_config={
"model_id": "deepseek",
"model_source": "deepseek-ai/DeepSeek-R1",
},
deployment_config={
"autoscaling_config": {
"min_replicas": 1,
"max_replicas": 1,
}
},
# Change to the accelerator type of the node
accelerator_type="H100",
runtime_env={"env_vars": {"VLLM_USE_V1": "1"}},
# Customize engine arguments as needed (e.g. vLLM engine kwargs)
engine_kwargs={
"tensor_parallel_size": 8,
"pipeline_parallel_size": 2,
"gpu_memory_utilization": 0.92,
"dtype": "auto",
"max_num_seqs": 40,
"max_model_len": 16384,
"enable_chunked_prefill": True,
"enable_prefix_caching": True,
},
)
# Deploy the application
llm_app = build_openai_app({"llm_configs": [llm_config]})
serve.run(llm_app)
```
### Production Deployment
For production deployments, save the following to a YAML file named `deepseek.yaml` and run `serve run deepseek.yaml`.
```yaml
applications:
- args:
llm_configs:
- model_loading_config:
model_id: "deepseek"
model_source: "deepseek-ai/DeepSeek-R1"
accelerator_type: "H100"
deployment_config:
autoscaling_config:
min_replicas: 1
max_replicas: 1
runtime_env:
env_vars:
VLLM_USE_V1: "1"
engine_kwargs:
tensor_parallel_size: 8
pipeline_parallel_size: 2
gpu_memory_utilization: 0.92
dtype: "auto"
max_num_seqs: 40
max_model_len: 16384
enable_chunked_prefill: true
enable_prefix_caching: true
import_path: ray.serve.llm:build_openai_app
name: llm_app
route_prefix: "/"
```
## Configuration
You may need to adjust configurations in the above code based on your setup, specifically:
* `accelerator_type`: for NVIDIA GPUs, DeepSeek requires Hopper GPUs or later ones. Therefore, you can specify `H200`, `H100`, `H20` etc. based on your hardware.
* `tensor_parallel_size` and `pipeline_parallel_size`: DeepSeek requires a single node of 8xH200, or two nodes of 8xH100. The typical setup of using H100 is setting `tensor_parallel_size` to `8` and `pipeline_parallel_size` to `2` as in the code example. When using H200, you can set `tensor_parallel_size` to `8` and leave out the `pipeline_parallel_size` parameter (it is `1` by default).
* `model_source`: although you could specify a HuggingFace model ID like `deepseek-ai/DeepSeek-R1` in the code example, it is recommended to pre-download the model because it is huge. You can download it to the local file system (e.g., `/path/to/downloaded/model`) or to a remote object store (e.g., `s3://my-bucket/path/to/downloaded/model`), and specify it as `model_source`. It is recommended to download it to a remote object store, using {ref}`Ray model caching utilities <model_cache>`. Note that if you have two nodes and would like to download to local file system, you need to download the model to the same path on both nodes.
## Testing the Service
You can query the deployed model using the following request and get the corresponding response.
::::{tab-set}
:::{tab-item} Request
```bash
curl -X POST http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer fake-key" \
-d '{
"model": "deepseek",
"messages": [{"role": "user", "content": "Hello!"}]
}'
```
:::
:::{tab-item} Response
```bash
{"id":"deepseek-68b5d5c5-fd34-42fc-be26-0a36f8457ffe","object":"chat.completion","created":1743646776,"model":"deepseek","choices":[{"index":0,"message":{"role":"assistant","reasoning_content":null,"content":"Hello! How can I assist you today? 😊","tool_calls":[]},"logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":6,"total_tokens":18,"completion_tokens":12,"prompt_tokens_details":null},"prompt_logprobs":null}
```
:::
::::
Another example request and response:
::::{tab-set}
:::{tab-item} Request
```bash
curl -X POST http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer fake-key" \
-d '{
"model": "deepseek",
"messages": [{"role": "user", "content": "The future of AI is"}]
}'
```
:::
:::{tab-item} Response
```bash
{"id":"deepseek-b81ff9be-3ffc-4811-80ff-225006eff27c","object":"chat.completion","created":1743646860,"model":"deepseek","choices":[{"index":0,"message":{"role":"assistant","reasoning_content":null,"content":"The future of AI is multifaceted and holds immense potential across various domains. Here are some key aspects that are likely to shape its trajectory:\n\n1. **Advanced Automation**: AI will continue to automate routine and complex tasks across industries, increasing efficiency and productivity. This includes everything from manufacturing and logistics to healthcare and finance.\n\n2. **Enhanced Decision-Making**: AI systems will provide deeper insights and predictive analytics, aiding in better decision-making processes for businesses, governments, and individuals.\n\n3. **Personalization**: AI will drive more personalized experiences in areas such as shopping, education, and entertainment, tailoring services and products to individual preferences and behaviors.\n\n4. **Healthcare Revolution**: AI will play a significant role in diagnosing diseases, personalizing treatment plans, and even predicting health issues before they become critical, potentially transforming the healthcare industry.\n\n5. **Ethical and Responsible AI**: As AI becomes more integrated into society, there will be a growing focus on developing ethical guidelines and frameworks to ensure AI is used responsibly and transparently, addressing issues like bias, privacy, and security.\n\n6. **Human-AI Collaboration**: The future will see more seamless collaboration between humans and AI, with AI augmenting human capabilities rather than replacing them. This includes areas like creative industries, where AI can assist in generating ideas and content.\n\n7. **AI in Education**: AI will personalize learning experiences, adapt to individual learning styles, and provide real-time feedback, making education more accessible and effective.\n\n8. **Robotics and Autonomous Systems**: Advances in AI will lead to more sophisticated robots and autonomous systems, impacting industries like transportation (e.g., self-driving cars), agriculture, and home automation.\n\n9. **AI and Sustainability**: AI will play a crucial role in addressing environmental challenges by optimizing resource use, improving energy efficiency, and aiding in climate modeling and conservation efforts.\n\n10. **Regulation and Governance**: As AI technologies advance, there will be increased efforts to establish international standards and regulations to govern their development and use, ensuring they benefit society as a whole.\n\n11. **Quantum Computing and AI**: The integration of quantum computing with AI could revolutionize data processing capabilities, enabling the solving of complex problems that are currently intractable.\n\n12. **AI in Creative Fields**: AI will continue to make strides in creative domains such as music, art, and literature, collaborating with human creators to push the boundaries of innovation and expression.\n\nOverall, the future of AI is both promising and challenging, requiring careful consideration of its societal impact and the ethical implications of its widespread adoption.","tool_calls":[]},"logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":9,"total_tokens":518,"completion_tokens":509,"prompt_tokens_details":null},"prompt_logprobs":null}
```
:::
::::
## Deploying with KubeRay
Create a KubeRay cluster using the {ref}`Ray Serve LLM KubeRay guide <kuberay-rayservice-llm-example>` with sufficient GPU resources for DeepSeek R1. For example, two 8xH100 nodes.
Deploy DeepSeek-R1 as a RayService with the following configuration file:
```bash
kubectl apply -f https://raw.githubusercontent.com/ray-project/kuberay/master/ray-operator/config/samples/ray-service.deepseek.yaml
```
## Troubleshooting
### Multi-Node GPU Issues
Since DeepSeek typically requires multi-node GPU deployment, you may encounter issues specific to multi-node GPU serving. Common problems include:
* **NCCL initialization failures**: Especially on H100 instances due to outdated `aws-ofi-plugin` versions
* **Pipeline parallelism hangs**: When `pipeline_parallel_size > 1`, the model serving may hang due to resource conflicts
For comprehensive troubleshooting of multi-node GPU serving issues, refer to {ref}`Troubleshooting multi-node GPU serving on KubeRay <serve-multi-node-gpu-troubleshooting>`.
doc/source/serve/tutorials/serve-ml-models.md
+260
View File
@@ -0,0 +1,260 @@
---
orphan: true
---
(serve-ml-models-tutorial)=
# Serve ML Models (Tensorflow, PyTorch, Scikit-Learn, others)
This guide shows how to train models from various machine learning frameworks and deploy them to Ray Serve.
See the [Key Concepts](serve-key-concepts) to learn more general information about Ray Serve.
:::::{tab-set}
::::{tab-item} Keras and TensorFlow
This example trains and deploys a simple TensorFlow neural net. In particular, it shows:
- How to train a TensorFlow model and load the model from your file system in your Ray Serve deployment.
- How to parse the JSON request and make a prediction.
Ray Serve is framework-agnostic--you can use any version of TensorFlow. This tutorial uses TensorFlow 2 and Keras. You also need `requests` to send HTTP requests to your model deployment. If you haven't already, install TensorFlow 2 and requests by running:
```console
$ pip install "tensorflow>=2.0" requests "ray[serve]"
```
Open a new Python file called `tutorial_tensorflow.py`. First, import Ray Serve and some other helpers.
```{literalinclude} ../doc_code/tutorial_tensorflow.py
:start-after: __doc_import_begin__
:end-before: __doc_import_end__
```
Next, train a simple MNIST model using Keras.
```{literalinclude} ../doc_code/tutorial_tensorflow.py
:start-after: __doc_train_model_begin__
:end-before: __doc_train_model_end__
```
Next, define a `TFMnistModel` class that accepts HTTP requests and runs the MNIST model that you trained. The `@serve.deployment` decorator makes it a deployment object that you can deploy onto Ray Serve. Note that Ray Serve exposes the deployment over an HTTP route. By default, when the deployment receives a request over HTTP, Ray Serve invokes the `__call__` method.
```{literalinclude} ../doc_code/tutorial_tensorflow.py
:start-after: __doc_define_servable_begin__
:end-before: __doc_define_servable_end__
```
:::{note}
When you deploy and instantiate the `TFMnistModel` class, Ray Serve loads the TensorFlow model from your file system so that it can be ready to run inference on the model and serve requests later.
:::
Now that you've defined the Serve deployment, prepare it so that you can deploy it.
```{literalinclude} ../doc_code/tutorial_tensorflow.py
:start-after: __doc_deploy_begin__
:end-before: __doc_deploy_end__
```
:::{note}
`TFMnistModel.bind(TRAINED_MODEL_PATH)` binds the argument `TRAINED_MODEL_PATH` to the deployment and returns a `DeploymentNode` object, a wrapping of the `TFMnistModel` deployment object, that you can then use to connect with other `DeploymentNodes` to form a more complex [deployment graph](serve-model-composition).
:::
Finally, deploy the model to Ray Serve through the terminal.
```console
$ serve run tutorial_tensorflow:mnist_model
```
Next, query the model. While Serve is running, open a separate terminal window, and run the following in an interactive Python shell or a separate Python script:
```python
import requests
import numpy as np
resp = requests.get(
"http://localhost:8000/", json={"array": np.random.randn(28 * 28).tolist()}
)
print(resp.json())
```
You should get an output like the following, although the exact prediction may vary:
```bash
{
"prediction": [[-1.504277229309082, ..., -6.793371200561523]],
"file": "/tmp/mnist_model.h5"
}
```
::::
::::{tab-item} PyTorch
This example loads and deploys a PyTorch ResNet model. In particular, it shows:
- How to load the model from PyTorch's pre-trained Model Zoo.
- How to parse the JSON request, transform the payload and make a prediction.
This tutorial requires PyTorch and Torchvision. Ray Serve is framework agnostic and works with any version of PyTorch. You also need `requests` to send HTTP requests to your model deployment. If you haven't already, install them by running:
```console
$ pip install torch torchvision requests "ray[serve]"
```
Open a new Python file called `tutorial_pytorch.py`. First, import Ray Serve and some other helpers.
```{literalinclude} ../doc_code/tutorial_pytorch.py
:start-after: __doc_import_begin__
:end-before: __doc_import_end__
```
Define a class `ImageModel` that parses the input data, transforms the images, and runs the ResNet18 model loaded from `torchvision`. The `@serve.deployment` decorator makes it a deployment object that you can deploy onto Ray Serve. Note that Ray Serve exposes the deployment over an HTTP route. By default, when the deployment receives a request over HTTP, Ray Serve invokes the `__call__` method.
```{literalinclude} ../doc_code/tutorial_pytorch.py
:start-after: __doc_define_servable_begin__
:end-before: __doc_define_servable_end__
```
:::{note}
When you deploy and instantiate an `ImageModel` class, Ray Serve loads the ResNet18 model from `torchvision` so that it can be ready to run inference on the model and serve requests later.
:::
Now that you've defined the Serve deployment, prepare it so that you can deploy it.
```{literalinclude} ../doc_code/tutorial_pytorch.py
:start-after: __doc_deploy_begin__
:end-before: __doc_deploy_end__
```
:::{note}
`ImageModel.bind()` returns a `DeploymentNode` object, a wrapping of the `ImageModel` deployment object, that you can then use to connect with other `DeploymentNodes` to form a more complex [deployment graph](serve-model-composition).
:::
Finally, deploy the model to Ray Serve through the terminal.
```console
$ serve run tutorial_pytorch:image_model
```
Next, query the model. While Serve is running, open a separate terminal window, and run the following in an interactive Python shell or a separate Python script:
```python
import requests
ray_logo_bytes = requests.get(
"https://raw.githubusercontent.com/ray-project/"
"ray/master/doc/source/images/ray_header_logo.png"
).content
resp = requests.post("http://localhost:8000/", data=ray_logo_bytes)
print(resp.json())
```
You should get an output like the following, although the exact number may vary:
```bash
{'class_index': 919}
```
::::
::::{tab-item} Scikit-learn
This example trains and deploys a simple scikit-learn classifier. In particular, it shows:
- How to load the scikit-learn model from file system in your Ray Serve definition.
- How to parse the JSON request and make a prediction.
Ray Serve is framework-agnostic. You can use any version of sklearn. You also need `requests` to send HTTP requests to your model deployment. If you haven't already, install scikit-learn and requests by running:
```console
$ pip install scikit-learn requests "ray[serve]"
```
Open a new Python file called `tutorial_sklearn.py`. Import Ray Serve and some other helpers.
```{literalinclude} ../doc_code/tutorial_sklearn.py
:start-after: __doc_import_begin__
:end-before: __doc_import_end__
```
**Train a Classifier**
Next, train a classifier with the [Iris dataset](https://scikit-learn.org/stable/auto_examples/datasets/plot_iris_dataset.html).
First, instantiate a `GradientBoostingClassifier` loaded from scikit-learn.
```{literalinclude} ../doc_code/tutorial_sklearn.py
:start-after: __doc_instantiate_model_begin__
:end-before: __doc_instantiate_model_end__
```
Next, load the Iris dataset and split the data into training and validation sets.
```{literalinclude} ../doc_code/tutorial_sklearn.py
:start-after: __doc_data_begin__
:end-before: __doc_data_end__
```
Then, train the model and save it to a file.
```{literalinclude} ../doc_code/tutorial_sklearn.py
:start-after: __doc_train_model_begin__
:end-before: __doc_train_model_end__
```
**Deploy with Ray Serve**
Finally, you're ready to deploy the classifier using Ray Serve.
Define a `BoostingModel` class that runs inference on the `GradientBoosingClassifier` model you trained and returns the resulting label. It's decorated with `@serve.deployment` to make it a deployment object so you can deploy it onto Ray Serve. Note that Ray Serve exposes the deployment over an HTTP route. By default, when the deployment receives a request over HTTP, Ray Serve invokes the `__call__` method.
```{literalinclude} ../doc_code/tutorial_sklearn.py
:start-after: __doc_define_servable_begin__
:end-before: __doc_define_servable_end__
```
:::{note}
When you deploy and instantiate a `BoostingModel` class, Ray Serve loads the classifier model that you trained from the file system so that it can be ready to run inference on the model and serve requests later.
:::
After you've defined the Serve deployment, prepare it so that you can deploy it.
```{literalinclude} ../doc_code/tutorial_sklearn.py
:start-after: __doc_deploy_begin__
:end-before: __doc_deploy_end__
```
:::{note}
`BoostingModel.bind(MODEL_PATH, LABEL_PATH)` binds the arguments `MODEL_PATH` and `LABEL_PATH` to the deployment and returns a `DeploymentNode` object, a wrapping of the `BoostingModel` deployment object, that you can then use to connect with other `DeploymentNodes` to form a more complex [deployment graph](serve-model-composition).
:::
Finally, deploy the model to Ray Serve through the terminal.
```console
$ serve run tutorial_sklearn:boosting_model
```
Next, query the model. While Serve is running, open a separate terminal window, and run the following in an interactive Python shell or a separate Python script:
```python
import requests
sample_request_input = {
"sepal length": 1.2,
"sepal width": 1.0,
"petal length": 1.1,
"petal width": 0.9,
}
response = requests.get("http://localhost:8000/", json=sample_request_input)
print(response.text)
```
You should get an output like the following, although the exact prediction may vary:
```python
{"result": "versicolor"}
```
::::
:::::
doc/source/serve/tutorials/stable-diffusion.md
+56
View File
@@ -0,0 +1,56 @@
---
orphan: true
---
(serve-stable-diffusion-tutorial)=
# Serve a Stable Diffusion Model
<a href="https://https://console.anyscale.com/register/ha?render_flow=ray&utm_source=ray_docs&utm_medium=docs&utm_campaign=ray-serve-stable-diffusion-quickstart&redirectTo=/v2/template-preview/serve-stable-diffusion-v2">
<img src="../../_static/img/run-on-anyscale.svg" alt="Run on Anyscale">
</a>
<br></br>
This example runs a Stable Diffusion application with Ray Serve.
To run this example, install the following:
```bash
pip install "ray[serve]" requests torch diffusers==0.35.2 transformers
```
This example uses the [stabilityai/stable-diffusion-xl-base-1.0](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0) model and [FastAPI](https://fastapi.tiangolo.com/) to build the example. Save the following code to a file named stable_diffusion.py.
The Serve code is as follows:
```{literalinclude} ../doc_code/stable_diffusion.py
:language: python
:start-after: __example_code_start__
:end-before: __example_code_end__
```
Use `serve run stable_diffusion:entrypoint` to start the Serve application.
:::{note}
The autoscaling config sets `min_replicas` to 0, which means the deployment starts with no `ObjectDetection` replicas. These replicas spawn only when a request arrives. When no requests arrive after a certain period of time, Serve downscales `ObjectDetection` back to 0 replica to save GPU resources.
:::
You should see these messages in the output:
```text
(ServeController pid=362, ip=10.0.44.233) INFO 2023-03-08 16:44:57,579 controller 362 http_state.py:129 - Starting HTTP proxy with name 'SERVE_CONTROLLER_ACTOR:SERVE_PROXY_ACTOR-7396d5a9efdb59ee01b7befba448433f6c6fc734cfa5421d415da1b3' on node '7396d5a9efdb59ee01b7befba448433f6c6fc734cfa5421d415da1b3' listening on '127.0.0.1:8000'
(ServeController pid=362, ip=10.0.44.233) INFO 2023-03-08 16:44:57,588 controller 362 http_state.py:129 - Starting HTTP proxy with name 'SERVE_CONTROLLER_ACTOR:SERVE_PROXY_ACTOR-a30ea53938547e0bf88ce8672e578f0067be26a7e26d23465c46300b' on node 'a30ea53938547e0bf88ce8672e578f0067be26a7e26d23465c46300b' listening on '127.0.0.1:8000'
(ProxyActor pid=439, ip=10.0.44.233) INFO: Started server process [439]
(ProxyActor pid=5779) INFO: Started server process [5779]
(ServeController pid=362, ip=10.0.44.233) INFO 2023-03-08 16:44:59,362 controller 362 deployment_state.py:1333 - Adding 1 replica to deployment 'APIIngress'.
2023-03-08 16:45:01,316 SUCC <string>:93 -- Deployed Serve app successfully.
```
Use the following code to send requests:
```python
import requests
prompt = "a cute cat is dancing on the grass."
input = "%20".join(prompt.split(" "))
resp = requests.get(f"http://127.0.0.1:8000/imagine?prompt={input}")
with open("output.png", 'wb') as f:
f.write(resp.content)
```
The app saves the `output.png` file locally. The following is an example of an output image. ![image](https://raw.githubusercontent.com/ray-project/images/master/docs/serve/stable_diffusion_output.png)
doc/source/serve/tutorials/streaming.md
+206
View File
@@ -0,0 +1,206 @@
---
orphan: true
---
(serve-streaming-tutorial)=
# Serve a Chatbot with Request and Response Streaming
This example deploys a chatbot that streams output back to the user. It shows:
* How to stream outputs from a Serve application
* How to use WebSockets in a Serve application
* How to combine batching requests with streaming outputs
This tutorial should help you with following use cases:
* You want to serve a large language model and stream results back token-by-token.
* You want to serve a chatbot that accepts a stream of inputs from the user.
This tutorial serves the [DialoGPT](https://huggingface.co/microsoft/DialoGPT-small) language model. Install the Hugging Face library to access it:
```
pip install "ray[serve]" transformers torch
```
## Create a streaming deployment
Open a new Python file called `textbot.py`. First, add the imports and the [Serve logger](serve-logging).
```{literalinclude} ../doc_code/streaming_tutorial.py
:language: python
:start-after: __textbot_setup_start__
:end-before: __textbot_setup_end__
```
Create a [FastAPI deployment](serve-fastapi-http), and initialize the model and the tokenizer in the constructor:
```{literalinclude} ../doc_code/streaming_tutorial.py
:language: python
:start-after: __textbot_constructor_start__
:end-before: __textbot_constructor_end__
```
Note that the constructor also caches an `asyncio` loop. This behavior is useful when you need to run a model and concurrently stream its tokens back to the user.
Add the following logic to handle requests sent to the `Textbot`:
```{literalinclude} ../doc_code/streaming_tutorial.py
:language: python
:start-after: __textbot_logic_start__
:end-before: __textbot_logic_end__
```
`Textbot` uses three methods to handle requests:
* `handle_request`: the entrypoint for HTTP requests. FastAPI automatically unpacks the `prompt` query parameter and passes it into `handle_request`. This method then creates a `TextIteratorStreamer`. Hugging Face provides this streamer as a convenient interface to access tokens generated by a language model. `handle_request` then kicks off the model in a background thread using `self.loop.run_in_executor`. This behavior lets the model generate tokens while `handle_request` concurrently calls `self.consume_streamer` to stream the tokens back to the user. `self.consume_streamer` is a generator that yields tokens one by one from the streamer. Lastly, `handle_request` passes the `self.consume_streamer` generator into a Starlette `StreamingResponse` and returns the response. Serve unpacks the Starlette `StreamingResponse` and yields the contents of the generator back to the user one by one.
* `generate_text`: the method that runs the model. This method runs in a background thread kicked off by `handle_request`. It pushes generated tokens into the streamer constructed by `handle_request`.
* `consume_streamer`: a generator method that consumes the streamer constructed by `handle_request`. This method keeps yielding tokens from the streamer until the model in `generate_text` closes the streamer. This method avoids blocking the event loop by calling `asyncio.sleep` with a brief timeout whenever the streamer is empty and waiting for a new token.
Bind the `Textbot` to a language model. For this tutorial, use the `"microsoft/DialoGPT-small"` model:
```{literalinclude} ../doc_code/streaming_tutorial.py
:language: python
:start-after: __textbot_bind_start__
:end-before: __textbot_bind_end__
```
Run the model with `serve run textbot:app`, and query it from another terminal window with this script:
```{literalinclude} ../doc_code/streaming_tutorial.py
:language: python
:start-after: __stream_client_start__
:end-before: __stream_client_end__
```
You should see the output printed token by token.
## Stream inputs and outputs using WebSockets
WebSockets let you stream input into the application and stream output back to the client. Use WebSockets to create a chatbot that stores a conversation with a user.
Create a Python file called `chatbot.py`. First add the imports:
```{literalinclude} ../doc_code/streaming_tutorial.py
:language: python
:start-after: __chatbot_setup_start__
:end-before: __chatbot_setup_end__
```
Create a FastAPI deployment, and initialize the model and the tokenizer in the constructor:
```{literalinclude} ../doc_code/streaming_tutorial.py
:language: python
:start-after: __chatbot_constructor_start__
:end-before: __chatbot_constructor_end__
```
Add the following logic to handle requests sent to the `Chatbot`:
```{literalinclude} ../doc_code/streaming_tutorial.py
:language: python
:start-after: __chatbot_logic_start__
:end-before: __chatbot_logic_end__
```
The `generate_text` and `consume_streamer` methods are the same as they were for the `Textbot`. The `handle_request` method has been updated to handle WebSocket requests.
The `handle_request` method is decorated with a `fastapi_app.websocket` decorator, which lets it accept WebSocket requests. First it `awaits` to accept the client's WebSocket request. Then, until the client disconnects, it does the following:
* gets the prompt from the client with `ws.receive_text`
* starts a new `TextIteratorStreamer` to access generated tokens
* runs the model in a background thread on the conversation so far
* streams the model's output back using `ws.send_text`
* stores the prompt and the response in the `conversation` string
Each time `handle_request` gets a new prompt from a client, it runs the whole conversation–with the new prompt appended–through the model. When the model finishes generating tokens, `handle_request` sends the `"<<Response Finished>>"` string to inform the client that the model has generated all tokens. `handle_request` continues to run until the client explicitly disconnects. This disconnect raises a `WebSocketDisconnect` exception, which ends the call.
Read more about WebSockets in the [FastAPI documentation](https://fastapi.tiangolo.com/advanced/websockets/).
Bind the `Chatbot` to a language model. For this tutorial, use the `"microsoft/DialoGPT-small"` model:
```{literalinclude} ../doc_code/streaming_tutorial.py
:language: python
:start-after: __chatbot_bind_start__
:end-before: __chatbot_bind_end__
```
Run the model with `serve run chatbot:app`. Query it using the `websockets` package, using `pip install websockets`:
```{literalinclude} ../doc_code/streaming_tutorial.py
:language: python
:start-after: __ws_client_start__
:end-before: __ws_client_end__
```
You should see the outputs printed token by token.
## Batch requests and stream the output for each
Improve model utilization and request latency by batching requests together when running the model.
Create a Python file called `batchbot.py`. First add the imports:
```{literalinclude} ../doc_code/streaming_tutorial.py
:language: python
:start-after: __batchbot_setup_start__
:end-before: __batchbot_setup_end__
```
:::{warning}
Hugging Face's support for `Streamers` is still under development and may change in the future. `RawQueue` is compatible with the `Streamers` interface in Hugging Face 4.30.2. However, the `Streamers` interface may change, making the `RawQueue` incompatible with Hugging Face models in the future.
:::
Similar to `Textbot` and `Chatbot`, the `Batchbot` needs a streamer to stream outputs from batched requests, but Hugging Face `Streamers` don't support batched requests. Add this custom `RawStreamer` to process batches of tokens:
```{literalinclude} ../doc_code/streaming_tutorial.py
:language: python
:start-after: __raw_streamer_start__
:end-before: __raw_streamer_end__
```
Create a FastAPI deployment, and initialize the model and the tokenizer in the constructor:
```{literalinclude} ../doc_code/streaming_tutorial.py
:language: python
:start-after: __batchbot_constructor_start__
:end-before: __batchbot_constructor_end__
```
Unlike `Textbot` and `Chatbot`, the `Batchbot` constructor also sets a `pad_token`. You need to set this token to batch prompts with different lengths.
Add the following logic to handle requests sent to the `Batchbot`:
```{literalinclude} ../doc_code/streaming_tutorial.py
:language: python
:start-after: __batchbot_logic_start__
:end-before: __batchbot_logic_end__
```
`Batchbot` uses four methods to handle requests:
* `handle_request`: the entrypoint method. This method simply takes in the request's prompt and calls the `run_model` method on it. `run_model` is a generator method that also handles batching the requests. `handle_request` passes `run_model` into a Starlette `StreamingResponse` and returns the response, so the bot can stream generated tokens back to the client.
* `run_model`: a generator method that performs batching. Since `run_model` is decorated with `@serve.batch`, it automatically takes in a batch of prompts. See the [batching guide](serve-batch-tutorial) for more info. `run_model` creates a `RawStreamer` to access the generated tokens. It calls `generate_text` in a background thread, and passes in the `prompts` and the `streamer`, similar to the `Textbot`. Then it iterates through the `consume_streamer` generator, repeatedly yielding a batch of tokens generated by the model.
* `generate_text`: the method that runs the model. It's mostly the same as `generate_text` in `Textbot`, with two differences. First, it takes in and processes a batch of prompts instead of a single prompt. Second, it sets `padding=True`, so prompts with different lengths can be batched together.
* `consume_streamer`: a generator method that consumes the streamer constructed by `handle_request`. It's mostly the same as `consume_streamer` in `Textbot`, with one difference. It uses the `tokenizer` to decode the generated tokens. Usually, the Hugging Face streamer handles the decoding. Because this implementation uses the custom `RawStreamer`, `consume_streamer` must handle the decoding.
:::{tip}
Some inputs within a batch may generate fewer outputs than others. When a particular input has nothing left to yield, pass a `StopIteration` object into the output iterable to terminate that input's request. See [Streaming batched requests](serve-streaming-batched-requests-guide) for more details.
:::
Bind the `Batchbot` to a language model. For this tutorial, use the `"microsoft/DialoGPT-small"` model:
```{literalinclude} ../doc_code/streaming_tutorial.py
:language: python
:start-after: __batchbot_bind_start__
:end-before: __batchbot_bind_end__
```
Run the model with `serve run batchbot:app`. Query it from two other terminal windows with this script:
```{literalinclude} ../doc_code/streaming_tutorial.py
:language: python
:start-after: __stream_client_start__
:end-before: __stream_client_end__
```
You should see the output printed token by token in both windows.
doc/source/serve/tutorials/text-classification.md
+52
View File
@@ -0,0 +1,52 @@
---
orphan: true
---
(serve-text-classification-tutorial)=
# Serve a Text Classification Model
This example uses a DistilBERT model to build an IMDB review classification application with Ray Serve.
To run this example, install the following:
```bash
pip install "ray[serve]" requests torch transformers
```
This example uses the [distilbert-base-uncased](https://huggingface.co/docs/transformers/tasks/sequence_classification) model and [FastAPI](https://fastapi.tiangolo.com/). Save the following code to a file named distilbert_app.py:
Use the following Serve code:
```{literalinclude} ../doc_code/distilbert.py
:language: python
:start-after: __example_code_start__
:end-before: __example_code_end__
```
Use `serve run distilbert_app:entrypoint` to start the Serve application.
:::{note}
The autoscaling config sets `min_replicas` to 0, which means the deployment starts with no `ObjectDetection` replicas. These replicas spawn only when a request arrives. When no requests arrive after a certain period of time, Serve downscales `ObjectDetection` back to 0 replica to save GPU resources.
:::
You should see the following messages in the logs:
```text
(ServeController pid=362, ip=10.0.44.233) INFO 2023-03-08 16:44:57,579 controller 362 http_state.py:129 - Starting HTTP proxy with name 'SERVE_CONTROLLER_ACTOR:SERVE_PROXY_ACTOR-7396d5a9efdb59ee01b7befba448433f6c6fc734cfa5421d415da1b3' on node '7396d5a9efdb59ee01b7befba448433f6c6fc734cfa5421d415da1b3' listening on '127.0.0.1:8000'
(ServeController pid=362, ip=10.0.44.233) INFO 2023-03-08 16:44:57,588 controller 362 http_state.py:129 - Starting HTTP proxy with name 'SERVE_CONTROLLER_ACTOR:SERVE_PROXY_ACTOR-a30ea53938547e0bf88ce8672e578f0067be26a7e26d23465c46300b' on node 'a30ea53938547e0bf88ce8672e578f0067be26a7e26d23465c46300b' listening on '127.0.0.1:8000'
(ProxyActor pid=439, ip=10.0.44.233) INFO: Started server process [439]
(ProxyActor pid=5779) INFO: Started server process [5779]
(ServeController pid=362, ip=10.0.44.233) INFO 2023-03-08 16:44:59,362 controller 362 deployment_state.py:1333 - Adding 1 replica to deployment 'APIIngress'.
2023-03-08 16:45:01,316 SUCC <string>:93 -- Deployed Serve app successfully.
```
Use the following code to send requests:
```python
import requests
prompt = "This was a masterpiece. Not completely faithful to the books, but enthralling from beginning to end. Might be my favorite of the three."
input = "%20".join(prompt.split(" "))
resp = requests.get(f"http://127.0.0.1:8000/classify?sentence={prompt}")
print(resp.status_code, resp.json())
```
The output of the client code is the response status code, the label, which is positive in this example, and the label's score.
```text
200 [{'label': 'LABEL_1', 'score': 0.9994940757751465}]
```
doc/source/serve/tutorials/triton-server-integration.md
+172
View File
@@ -0,0 +1,172 @@
---
orphan: true
---
# Serving models with Triton Server in Ray Serve
This guide shows how to build an application with stable diffusion model using [NVIDIA Triton Server](https://github.com/triton-inference-server/server) in Ray Serve.
## Preparation
### Installation
It is recommended to use the `nvcr.io/nvidia/tritonserver:23.12-py3` image which already has the Triton Server python API library installed, and install the ray serve lib by `pip install "ray[serve]"` inside the image.
### Build and export a model
For this application, the encoder is exported to ONNX format and the stable diffusion model is exported to be TensorRT engine format which is being compatible with Triton Server. Here is the example to export models to be in ONNX format.
```python
import torch
from pathlib import Path
from diffusers import StableDiffusionPipeline
# Load a specific model version that's known to work well with ONNX conversion
model_id = "runwayml/stable-diffusion-v1-5" # This is often the most compatible
model_path = Path("model_repository/stable_diffusion/1")
device = "cuda" if torch.cuda.is_available() else "cpu"
pipe = StableDiffusionPipeline.from_pretrained(model_id)\
.to(device)
vae = pipe.vae
unet = pipe.unet
text_encoder = pipe.text_encoder
hidden_size = text_encoder.config.hidden_size
vae.forward = vae.decode
torch.onnx.export(
vae,
(torch.randn(1, 4, 64, 64), False),
"vae.onnx",
input_names=["latent_sample", "return_dict"],
output_names=["sample"],
dynamic_axes={
"latent_sample": {0: "batch", 1: "channels", 2: "height", 3: "width"},
},
do_constant_folding=True,
opset_version=14,
)
dummy_text_input = torch.ones((1, 77), dtype=torch.int64, device=device)
torch.onnx.export(
text_encoder,
dummy_text_input,
"encoder.onnx",
input_names=["input_ids"],
output_names=["last_hidden_state", "pooler_output"],
dynamic_axes={
"input_ids": {0: "batch", 1: "sequence"},
},
opset_version=14,
do_constant_folding=True,
)
```
From the script, the outputs are `vae.onnx` and `encoder.onnx`.
After the ONNX model exported, convert the ONNX model to the TensorRT engine serialized file. ([Details](https://github.com/NVIDIA/TensorRT/blob/release/9.2/samples/trtexec/README.md?plain=1#L22) about trtexec cli)
```bash
trtexec --onnx=vae.onnx --saveEngine=vae.plan --minShapes=latent_sample:1x4x64x64 --optShapes=latent_sample:4x4x64x64 --maxShapes=latent_sample:8x4x64x64 --fp16
```
### Prepare the model repository
Triton Server requires a [model repository](https://github.com/triton-inference-server/server/blob/main/docs/user_guide/model_repository.md) to store the models, which is a local directory or remote blob store (e.g. AWS S3) containing the model configuration and the model files. In our example, we will use a local directory as the model repository to save all the model files.
```bash
model_repo/
├── stable_diffusion
│   ├── 1
│   │   └── model.py
│   └── config.pbtxt
├── text_encoder
│   ├── 1
│   │   └── model.onnx
│   └── config.pbtxt
└── vae
├── 1
│   └── model.plan
└── config.pbtxt
```
The model repository contains three models: `stable_diffusion`, `text_encoder` and `vae`. Each model has a `config.pbtxt` file and a model file. The `config.pbtxt` file contains the model configuration, which is used to describe the model type and input/output formats.(you can learn more about model config file [here](https://github.com/triton-inference-server/server/blob/main/docs/user_guide/model_configuration.md)). To get config files for our example, you can download them from [here](https://github.com/triton-inference-server/tutorials/tree/main/Conceptual_Guide/Part_6-building_complex_pipelines/model_repository). We use `1` as the version of each model. The model files are saved in the version directory.
## Start the Triton Server inside a Ray Serve application
In each serve replica, there is a single Triton Server instance running. The API takes the model repository path as the parameter, and the Triton Serve instance is started during the replica initialization. The models can be loaded during the inference requests, and the loaded models are cached in the Triton Server instance.
Here is the inference code example for serving a model with Triton Server.([source](https://github.com/triton-inference-server/tutorials/blob/main/Triton_Inference_Server_Python_API/examples/rayserve/tritonserver_deployment.py))
```python
import numpy
import requests
import tritonserver
from fastapi import FastAPI
from PIL import Image
from ray import serve
app = FastAPI()
@serve.deployment(ray_actor_options={"num_gpus": 1})
@serve.ingress(app)
class TritonDeployment:
def __init__(self):
self._triton_server = tritonserver
# NOTE: Each worker node needs to have access to this directory.
# If you are using distributed multi-node setup, prefer to use
# remote storage like S3 to save the model repository and use it.
#
# If triton server is not able to access this location,
# the triton server will complain `failed to stat /workspace/models`.
model_repository = ["/workspace/models"]
self._triton_server = tritonserver.Server(
model_repository=model_repository,
model_control_mode=tritonserver.ModelControlMode.EXPLICIT,
log_info=False,
)
self._triton_server.start(wait_until_ready=True)
@app.get("/generate")
def generate(self, prompt: str, filename: str = "generated_image.jpg") -> None:
if not self._triton_server.model("stable_diffusion").ready():
try:
self._triton_server.load("text_encoder")
self._triton_server.load("vae")
self._stable_diffusion = self._triton_server.load("stable_diffusion")
if not self._stable_diffusion.ready():
raise Exception("Model not ready")
except Exception as error:
print(f"Error can't load stable diffusion model, {error}")
return
for response in self._stable_diffusion.infer(inputs={"prompt": [[prompt]]}):
generated_image = (
numpy.from_dlpack(response.outputs["generated_image"])
.squeeze()
.astype(numpy.uint8)
)
image_ = Image.fromarray(generated_image)
image_.save(filename)
if __name__ == "__main__":
# Deploy the deployment.
serve.run(TritonDeployment.bind())
# Query the deployment.
requests.get(
"http://localhost:8000/generate",
params={"prompt": "dogs in new york, realistic, 4k, photograph"},
)
```
Save the above code to a file named e.g. `triton_serve.py`, then run `python triton_serve.py` to start the server and send classify requests. After you run the above code, you should see the image generated `generated_image.jpg`. Check it out! ![image](https://raw.githubusercontent.com/ray-project/images/master/docs/serve/triton_server_stable_diffusion.jpg)
:::{note}
You can also use remote model repository, such as AWS S3, to store the model files. To use remote model repository, you need to set the `model_repository` variable to the remote model repository path. For example `model_repository = s3://<bucket_name>/<model_repository_path>`.
:::
If you find any bugs or have any suggestions, please let us know by [filing an issue](https://github.com/ray-project/ray/issues) on GitHub.
doc/source/serve/tutorials/video-analysis/README.ipynb
+1411
View File
File diff suppressed because it is too large Load Diff
doc/source/serve/tutorials/video-analysis/README.md
+724
View File
@@ -0,0 +1,724 @@
---
orphan: true
---
# Video analysis inference pipeline with Ray Serve
This notebook demonstrates how to build a production-grade video analysis pipeline with [Ray Serve](https://docs.ray.io/en/latest/serve/). The pipeline processes videos from S3 and extracts:
- **Tags**: [Zero-shot classification](https://huggingface.co/tasks/zero-shot-image-classification) labels (such as "kitchen", "office", "park")
- **Captions**: Retrieval-based descriptions matching the video content
- **Scene changes**: Detected transitions using [exponential moving average (EMA)](https://en.wikipedia.org/wiki/Exponential_smoothing) analysis
The system uses [**SigLIP**](https://huggingface.co/google/siglip-so400m-patch14-384) (`google/siglip-so400m-patch14-384`) as the vision-language backbone. SigLIP provides a unified embedding space for both images and text, enabling zero-shot classification and retrieval without task-specific fine-tuning.
## Architecture
The pipeline splits work across three [Ray Serve deployments](https://docs.ray.io/en/latest/serve/key-concepts.html#deployment), each optimized for its workload:
```
┌─────────────────────┐
┌───▶│ VideoEncoder │
│ │ (GPU) │
│ ├─────────────────────┤
┌─────────────────────┐ │ │ • SigLIP embedding │
│ VideoAnalyzer │──┤ │ • 16 frames/chunk │
│ (Ingress) │ │ │ • L2 normalization │
├─────────────────────┤ │ └─────────────────────┘
│ • S3 download │ │ num_gpus=1
│ • FFmpeg chunking │ │
│ • Request routing │ │ ┌─────────────────────┐
└─────────────────────┘ └───▶│ MultiDecoder │
num_cpus=6 │ (CPU) │
├─────────────────────┤
│ • Tag classification│
│ • Caption retrieval │
│ • Scene detection │
└─────────────────────┘
num_cpus=1
```
**Request lifecycle:**
1. `VideoAnalyzer` receives HTTP request with S3 video URI
2. Downloads video from S3, splits into fixed-duration chunks using FFmpeg
3. Sends all chunks to `VideoEncoder` concurrently
4. Encoder returns embedding references (stored in Ray object store)
5. `VideoAnalyzer` sends embeddings to `MultiDecoder` serially (for EMA state continuity)
6. Aggregates results and returns tags, captions, and scene changes
## Why Ray Serve?
This pipeline has three distinct workloads: a GPU-bound encoder running SigLIP, a CPU-bound decoder doing cosine similarity, and a CPU-heavy ingress running FFmpeg. Traditional serving frameworks force you to bundle these into a single container with fixed resources, wasting GPU when the decoder runs or starving FFmpeg when the encoder dominates.
Ray Serve solves this with **heterogeneous [resource allocation](https://docs.ray.io/en/latest/serve/configure-serve-deployment.html#configure-ray-serve-deployments) per deployment**. The encoder requests 1 GPU, the decoder requests 1 CPU, and the ingress requests 6 CPUs for parallel FFmpeg. Each deployment [scales independently](https://docs.ray.io/en/latest/serve/autoscaling-guide.html) based on its own queue depth—GPU replicas scale with encoding demand while CPU replicas scale separately with decoding demand. The load test demonstrates this: throughput scales near-linearly from 2.4 to 67.5 requests/second as the system provisions replicas to match load.
The pipeline also benefits from **zero-copy data transfer**. The ingress passes encoder results directly to the decoder as unawaited [DeploymentResponse](https://docs.ray.io/en/latest/serve/api/doc/ray.serve.handle.DeploymentResponse.html) references rather than serialized data. Ray stores the embeddings in its [object store](https://docs.ray.io/en/latest/ray-core/objects.html), and the decoder retrieves them directly without routing through the ingress. When encoder and decoder land on the same node, this transfer is zero-copy.
**Request pipelining** keeps the GPU saturated. By allowing two concurrent requests per encoder replica via `max_ongoing_requests`, one request prepares data on CPU while another computes on GPU. This achieves 100% GPU utilization without batching, which would add latency from waiting for requests to accumulate.
Finally, **[deployment composition](https://docs.ray.io/en/latest/serve/model_composition.html)** lets you define the encoder, decoder, and ingress as separate classes, then wire them together with `.bind()`. Ray Serve handles deployment ordering, health checks, and request routing. The ingress maintains explicit state (EMA for scene detection) across chunks, which works correctly even when autoscaling routes requests to different decoder replicas—no sticky sessions required.
---
## Setup
### Prerequisites
Before running this notebook, ensure you have:
| Requirement | Purpose |
|-------------|---------|
| **Pexels API key** | Download sample video (free at https://www.pexels.com/api/) |
| **S3 bucket** | Store videos and text embeddings |
| **AWS credentials** | Read/write access to your S3 bucket |
| **ffmpeg** | Video processing and frame extraction |
| **GPU** | Run SigLIP model for encoding (1 GPU minimum) |
Set these environment variables before running:
```bash
export PEXELS_API_KEY="your-pexels-api-key"
export S3_BUCKET="your-bucket-name"
export AWS_ACCESS_KEY_ID="..."
export AWS_SECRET_ACCESS_KEY="..."
```
> **Note on GPU type**: The benchmarks, design choices, and hyperparameters in this notebook were tuned for **NVIDIA L4 GPUs**. Different GPU types (A10G, T4, A100, etc.) have different memory bandwidth, compute throughput, and batch characteristics. You may need to adjust `max_ongoing_requests`, chunk sizes, and concurrency limits for optimal performance on other hardware.
```python
import os
PEXELS_API_KEY = os.environ.get("PEXELS_API_KEY") # Or set directly: "your-api-key"
S3_BUCKET = os.environ.get("S3_BUCKET") # Or set directly: "your-bucket"
```
### Download sample video
Before running the pipeline, we need a sample video in S3. This section downloads a video from Pexels, normalizes it, and uploads to S3.
**Why normalize videos?** We re-encode all videos to a consistent format:
- **384×384 resolution**: Matches SigLIP's input size exactly, eliminating resize during inference
- **30 fps**: Standardizes frame timing for consistent chunk boundaries
- **[H.264](https://en.wikipedia.org/wiki/Advanced_Video_Coding) codec (libx264)**: Fast seeking—FFmpeg can jump directly to any timestamp without decoding preceding frames. Some source codecs (VP9, HEVC) require sequential decoding, adding latency for chunk extraction
```python
import asyncio
try:
asyncio.get_running_loop()
except RuntimeError:
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
```
```python
from scripts.download_stock_videos import download_sample_videos
# Download sample videos (checks for existing manifest first, skips Pexels API if found)
S3_PREFIX = "anyscale-example/stock-videos/"
video_paths = asyncio.get_event_loop().run_until_complete(download_sample_videos(
api_key=PEXELS_API_KEY,
bucket=S3_BUCKET,
total=1, # Just need one sample video
s3_prefix=S3_PREFIX,
overwrite=False
))
if not video_paths:
raise RuntimeError("No videos downloaded")
SAMPLE_VIDEO_URI = video_paths[0]
print(f"\nSample video ready: {SAMPLE_VIDEO_URI}")
```
✅ S3 bucket 'anyscale-example-video-analysis-test-bucket' accessible ✅ Found existing manifest with 1 videos in S3 Skipping Pexels API download
Sample video ready: s3://anyscale-example-video-analysis-test-bucket/anyscale-example/stock-videos/kitchen_cooking_35395675_00.mp4
### Generate embeddings for text bank
The decoder matches video embeddings against precomputed **text embeddings** for tags and descriptions. We define the text banks here and use a [Ray task](https://docs.ray.io/en/latest/ray-core/tasks.html) to compute embeddings on GPU and upload to S3.
```python
from textbanks import TAGS, DESCRIPTIONS
print(f"Tags: {TAGS}")
print(f"Descriptions: {DESCRIPTIONS}")
```
Tags: ['kitchen', 'living room', 'office', 'meeting room', 'classroom', 'restaurant', 'cafe', 'grocery store', 'gym', 'warehouse', 'parking lot', 'city street', 'park', 'shopping mall', 'beach', 'sports field', 'hallway', 'lobby', 'bathroom', 'bedroom'] Descriptions: ['A person cooking in a kitchen', 'Someone preparing food on a counter', 'A chef working in a professional kitchen', 'People eating at a dining table', 'A group having a meal together', 'A person working at a desk', 'Someone typing on a laptop', 'A business meeting in progress', 'A presentation being given', 'People collaborating in an office', 'A teacher lecturing in a classroom', 'Students sitting at desks', 'A person giving a speech', 'Someone writing on a whiteboard', 'A customer shopping in a store', 'People browsing products on shelves', 'A cashier at a checkout counter', 'A person exercising at a gym', 'Someone lifting weights', 'A person running on a treadmill', 'People walking on a city sidewalk', 'Pedestrians crossing a street', 'Traffic moving through an intersection', 'Cars driving on a road', 'A vehicle parked in a lot', 'People walking through a park', 'Someone jogging outdoors', 'A group having a conversation', 'Two people talking face to face', 'A person on a phone call', 'Someone reading a book', 'A person watching television', 'People waiting in line', 'A crowded public space', 'An empty hallway or corridor', 'A person entering a building', 'Someone opening a door', 'A delivery being made', 'A person carrying boxes', 'Workers in a warehouse']
```python
import ray
from jobs.generate_text_embeddings import generate_embeddings_task
S3_EMBEDDINGS_PREFIX = "anyscale-example/embeddings/"
# Run the Ray task (uses TAGS and DESCRIPTIONS from textbanks module)
# Note: runtime_env ships local modules to worker nodes (job working_dir only applies to driver)
ray.init(runtime_env={"working_dir": "."}, ignore_reinit_error=True)
result = ray.get(generate_embeddings_task.remote(S3_BUCKET, S3_EMBEDDINGS_PREFIX))
print(f"\nText embeddings ready:")
print(f" Tags: {result['tag_embeddings']['s3_uri']}")
print(f" Descriptions: {result['description_embeddings']['s3_uri']}")
```
2026-01-06 08:27:08,101 INFO worker.py:1821 -- Connecting to existing Ray cluster at address: 10.0.45.10:6379... 2026-01-06 08:27:08,114 INFO worker.py:1998 -- Connected to Ray cluster. View the dashboard at https://session-lqu9h8iu3cpgv59j74p498djis.i.anyscaleuserdata-staging.com  2026-01-06 08:27:08,160 INFO packaging.py:463 -- Pushing file package 'gcs://_ray_pkg_56a868c6743600cdc03ad7fedef93ccaf9011e05.zip' (10.59MiB) to Ray cluster... 2026-01-06 08:27:08,200 INFO packaging.py:476 -- Successfully pushed file package 'gcs://_ray_pkg_56a868c6743600cdc03ad7fedef93ccaf9011e05.zip'. /home/ray/anaconda3/lib/python3.12/site-packages/ray/_private/worker.py:2046: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0 warnings.warn(
(generate_embeddings_task pid=6626, ip=10.0.222.50) ============================================================ (generate_embeddings_task pid=6626, ip=10.0.222.50) Starting text embedding generation (generate_embeddings_task pid=6626, ip=10.0.222.50) ============================================================ (generate_embeddings_task pid=6626, ip=10.0.222.50) (generate_embeddings_task pid=6626, ip=10.0.222.50) 📚 Loading text banks... (generate_embeddings_task pid=6626, ip=10.0.222.50) Tags: 20 (generate_embeddings_task pid=6626, ip=10.0.222.50) Descriptions: 40 (generate_embeddings_task pid=6626, ip=10.0.222.50) (generate_embeddings_task pid=6626, ip=10.0.222.50) 🤖 Loading SigLIP model: google/siglip-so400m-patch14-384 (generate_embeddings_task pid=6626, ip=10.0.222.50) Device: cuda
(generate_embeddings_task pid=6626, ip=10.0.222.50) Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
(generate_embeddings_task pid=6626, ip=10.0.222.50) Model loaded in 2.6s (generate_embeddings_task pid=6626, ip=10.0.222.50) (generate_embeddings_task pid=6626, ip=10.0.222.50) 🏷️ Generating tag embeddings... (generate_embeddings_task pid=6626, ip=10.0.222.50) Shape: (20, 1152) (generate_embeddings_task pid=6626, ip=10.0.222.50) Time: 0.21s (generate_embeddings_task pid=6626, ip=10.0.222.50) (generate_embeddings_task pid=6626, ip=10.0.222.50) 📝 Generating description embeddings... (generate_embeddings_task pid=6626, ip=10.0.222.50) Shape: (40, 1152) (generate_embeddings_task pid=6626, ip=10.0.222.50) Time: 0.25s (generate_embeddings_task pid=6626, ip=10.0.222.50) (generate_embeddings_task pid=6626, ip=10.0.222.50) ☁️ Uploading to S3 bucket: anyscale-example-video-analysis-test-bucket
Text embeddings ready: Tags: s3://anyscale-example-video-analysis-test-bucket/anyscale-example/embeddings/tag_embeddings.npz Descriptions: s3://anyscale-example-video-analysis-test-bucket/anyscale-example/embeddings/description_embeddings.npz
---
## Build the Ray Serve application
This section walks through building the video analysis pipeline step by step, introducing Ray Serve concepts as we go.
### GPU encoder
The `VideoEncoder` deployment runs on GPU and converts video frames to embeddings using SigLIP. Key configuration:
- `num_gpus=1`: Each replica requires a dedicated GPU
- `max_ongoing_requests=2`: Allows pipelining—while one request computes on GPU, another prepares data on CPU
**Why no request batching?** A single chunk (16 frames @ 384×384) already saturates GPU compute. Batching multiple requests would require holding them until a batch forms, adding latency without throughput gain. Instead, we use `max_ongoing_requests=2` to pipeline preparation and computation.
![GPU Utilization](assets/gpu_utilization.png)
**Why [`asyncio.to_thread`](https://docs.python.org/3/library/asyncio-task.html#asyncio.to_thread)?** Ray Serve deployments run in an async event loop. The `encode_frames` method is CPU/GPU-bound (PyTorch inference), which would block the event loop and prevent concurrent request handling. Wrapping it in `asyncio.to_thread` offloads the blocking work to a thread pool, keeping the event loop free to accept new requests.
**Why pass [`DeploymentResponse`](https://docs.ray.io/en/latest/serve/api/doc/ray.serve.handle.DeploymentResponse.html) to decoder?** Instead of awaiting the encoder result in `VideoAnalyzer` and passing raw data to the decoder, we pass the unawaited `DeploymentResponse` directly. Ray Serve automatically resolves this reference when the decoder needs it, storing the embeddings in the [object store](https://docs.ray.io/en/latest/ray-core/objects.html#objects-in-ray). This avoids an unnecessary serialize/deserialize round-trip through `VideoAnalyzer`—the decoder retrieves data directly from the object store, enabling zero-copy transfer if encoder and decoder are on the same node.
```python
from constants import MODEL_NAME
print(f"MODEL_NAME: {MODEL_NAME}")
```
(generate_embeddings_task pid=6626, ip=10.0.222.50) Tags: s3://anyscale-example-video-analysis-test-bucket/anyscale-example/embeddings/tag_embeddings.npz (generate_embeddings_task pid=6626, ip=10.0.222.50) Descriptions: s3://anyscale-example-video-analysis-test-bucket/anyscale-example/embeddings/description_embeddings.npz (generate_embeddings_task pid=6626, ip=10.0.222.50) (generate_embeddings_task pid=6626, ip=10.0.222.50) ✅ Done!
MODEL_NAME: google/siglip-so400m-patch14-384
```python
from deployments.encoder import VideoEncoder
import inspect
print(inspect.getsource(VideoEncoder.func_or_class))
```
@serve.deployment( num_replicas="auto", ray_actor_options={"num_gpus": 1, "num_cpus": 2}, # GPU utilization is at 100% when this is set to 2. with L4 # aka number on ongoing chunks that can be processed at once. max_ongoing_requests=2, autoscaling_config={ "min_replicas": 1, "max_replicas": 10, "target_num_ongoing_requests": 2, }, ) class VideoEncoder: """ Encodes video frames into embeddings using SigLIP.
Returns both per-frame embeddings and pooled embedding. """
def __init__(self): self.device = "cuda" if torch.cuda.is_available() else "cpu" print(f"VideoEncoder initializing on {self.device}")
# Load SigLIP model and processor self.processor = AutoProcessor.from_pretrained(MODEL_NAME) self.model = AutoModel.from_pretrained(MODEL_NAME).to(self.device) self.model.eval()
# Get embedding dimension self.embedding_dim = self.model.config.vision_config.hidden_size
print(f"VideoEncoder ready (embedding_dim={self.embedding_dim})")
def encode_frames(self, frames: np.ndarray) -> np.ndarray: """ Encode frames and return per-frame embeddings.
Args: frames: np.ndarray of shape (T, H, W, 3) uint8 RGB
Returns: np.ndarray of shape (T, D) float32, L2-normalized per-frame embeddings """
# Convert to PIL images pil_images = frames_to_pil_list(frames)
# Process images inputs = self.processor(images=pil_images, return_tensors="pt").to(self.device) # inputs = {k: v.to(self.device) for k, v in inputs.items()}
start = torch.cuda.Event(enable_timing=True) end = torch.cuda.Event(enable_timing=True)
# Get embeddings with torch.no_grad(): with torch.amp.autocast(device_type=self.device, enabled=self.device == "cuda"): outputs = self.model.get_image_features(**inputs)
# L2 normalize on GPU (faster than CPU numpy) frame_embeddings = torch.nn.functional.normalize(outputs, p=2, dim=1)
# Move to CPU and convert to numpy result = frame_embeddings.cpu().numpy().astype(np.float32) return result
async def encode_unbatched(self, frames: np.ndarray) -> dict: """ Unbatched entry point - processes single request directly.
Args: frames: np.ndarray of shape (T, H, W, 3)
Returns: dict with 'frame_embeddings' and 'embedding_dim' """ print(f"Unbatched: {frames.shape[0]} frames")
frame_embeddings = await asyncio.to_thread(self.encode_frames, frames)
return { "frame_embeddings": frame_embeddings, "embedding_dim": self.embedding_dim, }
@serve.batch(max_batch_size=2, batch_wait_timeout_s=0.1) async def encode_batched(self, frames_batch: List[np.ndarray]) -> List[dict]: """ Batched entry point - collects multiple requests into single GPU call.
Args: frames_batch: List of frame arrays, each of shape (T, H, W, 3)
Returns: List of dicts, each with 'frame_embeddings' and 'embedding_dim' """ frame_counts = [f.shape[0] for f in frames_batch] total_frames = sum(frame_counts)
print(f"Batched: {len(frames_batch)} requests ({total_frames} total frames)")
# Concatenate all frames into single batch all_frames = np.concatenate(frames_batch, axis=0)
# Single forward pass for all frames all_embeddings = await asyncio.to_thread(self.encode_frames, all_frames)
# Split results back per request results = [] offset = 0 for n_frames in frame_counts: chunk_embeddings = all_embeddings[offset:offset + n_frames] results.append({ "frame_embeddings": chunk_embeddings, "embedding_dim": self.embedding_dim, }) offset += n_frames
return results
async def __call__(self, frames: np.ndarray, use_batching: bool = False) -> dict: """ Main entry point. Set use_batching=False for direct comparison. """ if use_batching: return await self.encode_batched(frames) else: return await self.encode_unbatched(frames)
### CPU decoder
The `MultiDecoder` deployment runs on CPU and performs three tasks:
1. **Tag classification**: [Cosine similarity](https://en.wikipedia.org/wiki/Cosine_similarity) between video embedding and precomputed tag embeddings
2. **Caption retrieval**: Find the best-matching description from a text bank
3. **Scene detection**: EMA-based anomaly detection comparing each frame to recent history
The decoder loads precomputed text embeddings from S3 at startup.
**Why separate GPU/CPU deployments?** The encoder needs GPU for neural network inference; the decoder only does numpy dot products. Separating them allows independent scaling—encoders are limited by GPU count, decoders scale cheaply with CPU cores. This avoids tying expensive GPU resources to lightweight CPU work.
**Why EMA for scene detection?** Exponential Moving Average reuses existing SigLIP embeddings without an extra model. The algorithm computes `score = 1 - cosine(frame, EMA)` where EMA updates as `ema = 0.9*ema + 0.1*frame`. A simple threshold (`score > 0.15`) detects abrupt scene changes while smoothing noise.
```python
from deployments.decoder import MultiDecoder
print(inspect.getsource(MultiDecoder.func_or_class))
```
@serve.deployment( num_replicas="auto", ray_actor_options={"num_cpus": 1}, max_ongoing_requests=4, # can be set higher than 4, but since the encoder is limited to 4, we need to keep it at 4. autoscaling_config={ "min_replicas": 1, "max_replicas": 10, "target_num_ongoing_requests": 2, }, ) class MultiDecoder: """ Decodes video embeddings into tags, captions, and scene changes.
Uses precomputed text embeddings loaded from S3. This deployment is stateless - EMA state for scene detection is passed in and returned with each call, allowing the caller to maintain state continuity across multiple replicas. """
async def __init__(self, bucket: str, s3_prefix: str = S3_EMBEDDINGS_PREFIX): """Initialize decoder with text embeddings from S3.""" self.bucket = bucket self.ema_alpha = EMA_ALPHA self.scene_threshold = SCENE_CHANGE_THRESHOLD self.s3_prefix = s3_prefix logger.info(f"MultiDecoder initializing (bucket={self.bucket}, ema_alpha={self.ema_alpha}, threshold={self.scene_threshold})")
await self._load_embeddings()
logger.info(f"MultiDecoder ready (tags={len(self.tag_texts)}, descriptions={len(self.desc_texts)})")
async def _load_embeddings(self): """Load precomputed text embeddings from S3.""" session = aioboto3.Session(region_name=get_s3_region(self.bucket))
async with session.client("s3") as s3: # Load tag embeddings tag_key = f"{self.s3_prefix}tag_embeddings.npz" response = await s3.get_object(Bucket=self.bucket, Key=tag_key) tag_data = await response["Body"].read() tag_npz = np.load(io.BytesIO(tag_data), allow_pickle=True) self.tag_embeddings = tag_npz["embeddings"] self.tag_texts = tag_npz["texts"].tolist()
# Load description embeddings desc_key = f"{self.s3_prefix}description_embeddings.npz" response = await s3.get_object(Bucket=self.bucket, Key=desc_key) desc_data = await response["Body"].read() desc_npz = np.load(io.BytesIO(desc_data), allow_pickle=True) self.desc_embeddings = desc_npz["embeddings"] self.desc_texts = desc_npz["texts"].tolist()
def _cosine_similarity(self, embedding: np.ndarray, bank: np.ndarray) -> np.ndarray: """Compute cosine similarity between embedding and all vectors in bank.""" return bank @ embedding
def _get_top_tags(self, embedding: np.ndarray, top_k: int = 5) -> list[dict]: """Get top-k matching tags with scores.""" scores = self._cosine_similarity(embedding, self.tag_embeddings) top_indices = np.argsort(scores)[::-1][:top_k] return [ {"text": self.tag_texts[i], "score": float(scores[i])} for i in top_indices ]
def _get_retrieval_caption(self, embedding: np.ndarray) -> dict: """Get best matching description.""" scores = self._cosine_similarity(embedding, self.desc_embeddings) best_idx = np.argmax(scores) return { "text": self.desc_texts[best_idx], "score": float(scores[best_idx]), }
def _detect_scene_changes( self, frame_embeddings: np.ndarray, chunk_index: int, chunk_start_time: float, chunk_duration: float, ema_state: np.ndarray | None = None, ) -> tuple[list[dict], np.ndarray]: """ Detect scene changes using EMA-based scoring.
score_t = 1 - cosine(E_t, ema_t) ema_t = α * ema_{t-1} + (1-α) * E_t
Args: frame_embeddings: (T, D) normalized embeddings chunk_index: Index of this chunk in the video chunk_start_time: Start time of chunk in video (seconds) chunk_duration: Duration of chunk (seconds) ema_state: EMA state from previous chunk, or None for first chunk
Returns: Tuple of (scene_changes list, updated ema_state) """ num_frames = len(frame_embeddings) if num_frames == 0: # Return empty changes and unchanged state (or zeros if no state) return [], ema_state if ema_state is not None else np.zeros(0)
# Initialize EMA from first frame if no prior state ema = ema_state.copy() if ema_state is not None else frame_embeddings[0].copy() scene_changes = []
for frame_idx, embedding in enumerate(frame_embeddings): # Compute score: how different is current frame from recent history similarity = float(np.dot(embedding, ema)) score = max(0.0, 1.0 - similarity)
# Detect scene change if score exceeds threshold if score >= self.scene_threshold: # Calculate timestamp within video frame_offset = (frame_idx / max(1, num_frames - 1)) * chunk_duration timestamp = chunk_start_time + frame_offset
scene_changes.append({ "timestamp": round(timestamp, 3), "score": round(score, 4), "chunk_index": chunk_index, "frame_index": frame_idx, })
# Update EMA ema = self.ema_alpha * ema + (1 - self.ema_alpha) * embedding # Re-normalize ema = ema / np.linalg.norm(ema)
return scene_changes, ema
def __call__( self, encoder_output: dict, chunk_index: int, chunk_start_time: float, chunk_duration: float, top_k_tags: int = 5, ema_state: np.ndarray | None = None, ) -> dict: """ Decode embeddings into tags, caption, and scene changes.
Args: encoder_output: Dict with 'frame_embeddings' and 'embedding_dim' chunk_index: Index of this chunk in the video chunk_start_time: Start time of chunk (seconds) chunk_duration: Duration of chunk (seconds) top_k_tags: Number of top tags to return ema_state: EMA state from previous chunk for scene detection continuity. Pass None for the first chunk of a stream.
Returns: Dict containing tags, retrieval_caption, scene_changes, and updated ema_state. The caller should pass the returned ema_state to the next chunk's call. """ # Get frame embeddings from encoder output frame_embeddings = encoder_output["frame_embeddings"]
# Calculate pooled embedding (mean across frames, normalized) pooled_embedding = frame_embeddings.mean(axis=0) pooled_embedding = pooled_embedding / np.linalg.norm(pooled_embedding)
# Classification and retrieval on pooled embedding tags = self._get_top_tags(pooled_embedding, top_k=top_k_tags) caption = self._get_retrieval_caption(pooled_embedding)
# Scene change detection on frame embeddings scene_changes, new_ema_state = self._detect_scene_changes( frame_embeddings=frame_embeddings, chunk_index=chunk_index, chunk_start_time=chunk_start_time, chunk_duration=chunk_duration, ema_state=ema_state, )
return { "tags": tags, "retrieval_caption": caption, "scene_changes": scene_changes, "ema_state": new_ema_state, }
### Video chunking
Before we can process a video, we need to split it into fixed-duration chunks and extract frames. The chunking process:
1. **Get video duration** using [`ffprobe`](https://ffmpeg.org/ffprobe.html)
2. **Define chunk boundaries** (e.g., 0-10s, 10-20s, 20-30s for a 30s video)
3. **Extract frames in parallel** using multiple concurrent [`ffmpeg`](https://ffmpeg.org/) processes
4. **Limit concurrency** with [`asyncio.Semaphore`](https://docs.python.org/3/library/asyncio-sync.html#asyncio.Semaphore) to avoid CPU oversubscription
Each chunk extracts 16 frames uniformly sampled across its duration, resized to 384×384 (matching SigLIP's input size).
#### Design choices
**Direct S3 download vs presigned URLs**: We download the video to local disk before processing. An alternative is generating a presigned S3 URL and passing it directly to FFmpeg. Benchmarks show direct download is faster—FFmpeg's HTTP client doesn't handle S3's chunked responses as efficiently as `aioboto3`, and network latency compounds across multiple seeks.
![Presigned vs Direct S3](assets/presigned_vs_direct_s3.png)
**Single FFmpeg vs parallel FFmpeg**: Two approaches for extracting frames from multiple chunks:
- **Single FFmpeg**: One process reads the entire video, using `select` filter to pick frames at specific timestamps
- **Parallel FFmpeg**: Multiple concurrent processes, each extracting one chunk
Chose the Single FFmpeg approach since it outperforms parallel FFmpeg on longer videos and yields similar performance for typical 10s chunks. This method is both efficient and scalable as chunk counts grow.
![Single vs Multi FFmpeg](assets/single_vs_multi_ffmpeg.png)
**Chunk duration**: We use 10-second chunks. Shorter chunks increase overhead (more FFmpeg calls, more encoder/decoder round-trips). Longer chunks increases processing efficiency but **degrade inference quality**—SigLIP processes exactly 16 frames per chunk, so a 60-second chunk samples one frame every 3.75 seconds, missing fast scene changes. The 10-second sweet spot balances throughput with temporal resolution (~1.6 fps sampling).
![Chunk Video Analysis](assets/chunk_video_analysis.png)
### Deployment composition
The `VideoAnalyzer` ingress deployment orchestrates the encoder and decoder. It uses [FastAPI](https://fastapi.tiangolo.com/) integration with [`@serve.ingress`](https://docs.ray.io/en/latest/serve/http-guide.html#fastapi-http-deployments) for HTTP endpoints.
#### Design choices
**Why `num_cpus=6`?** The analyzer runs FFmpeg for frame extraction. Each FFmpeg process uses `FFMPEG_THREADS=2`, and we run up to `NUM_WORKERS=3` concurrent processes. So `2 × 3 = 6` CPUs ensures FFmpeg doesn't contend for CPU during parallel chunk extraction.
**Why `max_ongoing_requests=4`?** The encoder has `max_ongoing_requests=2`. We want the analyzer to stay ahead: while 2 videos are encoding, we download and chunk 2 more videos so they're ready when encoder slots free up. This keeps the GPU pipeline saturated without excessive memory from queued requests.
**Why cache the S3 client?** Creating a new `aioboto3` client per request adds overhead (connection setup, credential resolution). Caching the client in `__init__` and reusing it across requests amortizes this cost. The client is thread-safe and handles connection pooling internally.
**Why encode in parallel but decode serially?** Encoding is stateless—each chunk's frames go through SigLIP independently, so we fire all chunks concurrently with [`asyncio.gather`](https://docs.python.org/3/library/asyncio-task.html#asyncio.gather). Decoding requires temporal ordering—the EMA for scene detection must process chunks in order (chunk 0's EMA state feeds into chunk 1). The `VideoAnalyzer` calls the decoder serially, passing EMA state from each response to the next request. This explicit state passing ensures correct scene detection even when multiple decoder replicas exist under autoscaling.
```python
from app import VideoAnalyzer
print(inspect.getsource(VideoAnalyzer.func_or_class))
```
@serve.deployment( # setting this to twice that of the encoder. So that requests can complete the # upfront CPU work and be queued for GPU processing. num_replicas="auto", ray_actor_options={"num_cpus": FFMPEG_THREADS}, max_ongoing_requests=4, autoscaling_config={ "min_replicas": 2, "max_replicas": 20, "target_num_ongoing_requests": 2, }, ) @serve.ingress(fastapi_app) class VideoAnalyzer: """ Main ingress deployment that orchestrates VideoEncoder and MultiDecoder.
Encoder refs are passed directly to decoder; Ray Serve resolves dependencies. Downloads video from S3 to temp file for fast local processing. """
def __init__(self, encoder: VideoEncoder, decoder: MultiDecoder): self.encoder = encoder self.decoder = decoder self._s3_session = aioboto3.Session() self._s3_client = None # Cached client for reuse across requests logger.info("VideoAnalyzer ready")
async def _get_s3_client(self): """Get or create a reusable S3 client.""" if self._s3_client is None: self._s3_client = await self._s3_session.client("s3").__aenter__() return self._s3_client
async def _download_video(self, s3_uri: str) -> Path: """Download video from S3 to temp file. Returns local path.""" bucket, key = parse_s3_uri(s3_uri)
# Create temp file with video extension suffix = Path(key).suffix or ".mp4" temp_file = tempfile.NamedTemporaryFile(suffix=suffix, delete=False) temp_path = Path(temp_file.name) temp_file.close()
s3 = await self._get_s3_client() await s3.download_file(bucket, key, str(temp_path))
return temp_path
def _aggregate_results( self, chunk_results: list[dict], top_k_tags: int = 5, ) -> dict: """ Aggregate results from multiple chunks.
Strategy:
- Tags: Average scores across chunks, return top-k
- Caption: Return the one with highest score across all chunks """ # Aggregate tag scores tag_scores = defaultdict(list) for result in chunk_results: for tag in result["tags"]: tag_scores[tag["text"]].append(tag["score"])
# Average tag scores and sort aggregated_tags = [ {"text": text, "score": np.mean(scores)} for text, scores in tag_scores.items() ] aggregated_tags.sort(key=lambda x: x["score"], reverse=True) top_tags = aggregated_tags[:top_k_tags]
# Best caption across all chunks best_caption = max( (r["retrieval_caption"] for r in chunk_results), key=lambda x: x["score"], )
return { "tags": top_tags, "retrieval_caption": best_caption, }
def _encode_chunk(self, frames: np.ndarray, use_batching: bool = False) -> DeploymentResponse: """Encode a single chunk's frames to embeddings. Returns DeploymentResponse ref.""" return self.encoder.remote(frames, use_batching=use_batching)
async def _decode_chunk( self, encoder_output: dict, chunk_index: int, chunk_start_time: float, chunk_duration: float, ema_state=None, ) -> dict: """Decode embeddings to tags, caption, scene changes.""" return await self.decoder.remote( encoder_output=encoder_output, chunk_index=chunk_index, chunk_start_time=chunk_start_time, chunk_duration=chunk_duration, ema_state=ema_state, )
@fastapi_app.post("/analyze", response_model=AnalyzeResponse) async def analyze(self, request: AnalyzeRequest) -> AnalyzeResponse: """ Analyze a video from S3 and return tags, caption, and scene changes.
Downloads video to temp file for fast local processing. Chunks the entire video and aggregates results. Encoder refs are passed directly to decoder for dependency resolution. """ total_start = time.perf_counter() temp_path = None
try: # Download video from S3 to temp file download_start = time.perf_counter() try: temp_path = await self._download_video(request.video_path) except Exception as e: raise HTTPException(status_code=400, detail=f"Cannot download S3 video: {e}") s3_download_ms = (time.perf_counter() - download_start) * 1000
# Chunk video with PARALLEL frame extraction from local file decode_start = time.perf_counter() try: chunks = await chunk_video_async( str(temp_path), chunk_duration=request.chunk_duration, num_frames_per_chunk=request.num_frames, ffmpeg_threads=FFMPEG_THREADS, use_single_ffmpeg=True, ) except Exception as e: raise HTTPException(status_code=400, detail=f"Cannot process video: {e}")
decode_video_ms = (time.perf_counter() - decode_start) * 1000
if not chunks: raise HTTPException(status_code=400, detail="No chunks extracted from video")
# Calculate video duration from chunks video_duration = chunks[-1].start_time + chunks[-1].duration
# Fire off all encoder calls (returns refs, not awaited) encode_start = time.perf_counter() encode_refs = [ self._encode_chunk(chunk.frames, use_batching=request.use_batching) for chunk in chunks ] encode_ms = (time.perf_counter() - encode_start) * 1000
# Decode chunks SERIALLY, passing encoder refs directly. # Ray Serve resolves the encoder result when decoder needs it. # EMA state is tracked here (not in decoder) to ensure continuity # even when autoscaling routes requests to different replicas. decode_start = time.perf_counter() decode_results = [] ema_state = None # Will be initialized from first chunk's first frame for chunk, enc_ref in zip(chunks, encode_refs): dec_result = await self._decode_chunk( encoder_output=enc_ref, chunk_index=chunk.index, chunk_start_time=chunk.start_time, chunk_duration=chunk.duration, ema_state=ema_state, ) decode_results.append(dec_result) ema_state = dec_result["ema_state"] # Carry forward for next chunk decode_ms = (time.perf_counter() - decode_start) * 1000
# Collect results chunk_results = [] per_chunk_results = [] all_scene_changes = []
for chunk, decoder_result in zip(chunks, decode_results): chunk_results.append(decoder_result)
# Scene changes come directly from decoder chunk_scene_changes = [ SceneChange(**sc) for sc in decoder_result["scene_changes"] ] all_scene_changes.extend(chunk_scene_changes)
per_chunk_results.append(ChunkResult( chunk_index=chunk.index, start_time=chunk.start_time, duration=chunk.duration, tags=[TagResult(**t) for t in decoder_result["tags"]], retrieval_caption=CaptionResult(**decoder_result["retrieval_caption"]), scene_changes=chunk_scene_changes, ))
# Aggregate results aggregated = self._aggregate_results(chunk_results)
total_ms = (time.perf_counter() - total_start) * 1000
return AnalyzeResponse( stream_id=request.stream_id, tags=[TagResult(**t) for t in aggregated["tags"]], retrieval_caption=CaptionResult(**aggregated["retrieval_caption"]), scene_changes=all_scene_changes, num_scene_changes=len(all_scene_changes), chunks=per_chunk_results, num_chunks=len(chunks), video_duration=video_duration, timing_ms=TimingResult( s3_download_ms=round(s3_download_ms, 2), decode_video_ms=round(decode_video_ms, 2), encode_ms=round(encode_ms, 2), decode_ms=round(decode_ms, 2), total_ms=round(total_ms, 2), ), ) finally: # Clean up temp file if temp_path and temp_path.exists(): temp_path.unlink(missing_ok=True)
@fastapi_app.get("/health") async def health(self): """Health check endpoint.""" return {"status": "healthy"}
```python
encoder = VideoEncoder.bind()
decoder = MultiDecoder.bind(bucket=S3_BUCKET, s3_prefix=S3_EMBEDDINGS_PREFIX)
app = VideoAnalyzer.bind(encoder=encoder, decoder=decoder)
```
```python
from ray import serve
serve.run(app, name="video-analyzer", route_prefix="/", blocking=False)
```
(ProxyActor pid=251250) INFO 2026-01-06 08:27:23,294 proxy 10.0.45.10 -- Proxy starting on node 7beb9ebc3808dac027f36a0400592d7a27cc9e90f0aab0a2578c328b (HTTP port: 8000). INFO 2026-01-06 08:27:23,369 serve 250708 -- Started Serve in namespace "serve". (ServeController pid=251181) INFO 2026-01-06 08:27:23,394 controller 251181 -- Registering autoscaling state for deployment Deployment(name='VideoEncoder', app='video-analyzer') (ServeController pid=251181) INFO 2026-01-06 08:27:23,395 controller 251181 -- Deploying new version of Deployment(name='VideoEncoder', app='video-analyzer') (initial target replicas: 1). (ServeController pid=251181) INFO 2026-01-06 08:27:23,397 controller 251181 -- Registering autoscaling state for deployment Deployment(name='MultiDecoder', app='video-analyzer') (ServeController pid=251181) INFO 2026-01-06 08:27:23,397 controller 251181 -- Deploying new version of Deployment(name='MultiDecoder', app='video-analyzer') (initial target replicas: 1). (ServeController pid=251181) INFO 2026-01-06 08:27:23,398 controller 251181 -- Registering autoscaling state for deployment Deployment(name='VideoAnalyzer', app='video-analyzer') (ServeController pid=251181) INFO 2026-01-06 08:27:23,398 controller 251181 -- Deploying new version of Deployment(name='VideoAnalyzer', app='video-analyzer') (initial target replicas: 2). (ProxyActor pid=251250) INFO 2026-01-06 08:27:23,365 proxy 10.0.45.10 -- Got updated endpoints: {}. (ProxyActor pid=251250) INFO 2026-01-06 08:27:23,403 proxy 10.0.45.10 -- Got updated endpoints: {Deployment(name='VideoAnalyzer', app='video-analyzer'): EndpointInfo(route='/', app_is_cross_language=False, route_patterns=None)}. (ServeController pid=251181) INFO 2026-01-06 08:27:23,508 controller 251181 -- Adding 1 replica to Deployment(name='VideoEncoder', app='video-analyzer'). (ServeController pid=251181) INFO 2026-01-06 08:27:23,510 controller 251181 -- Adding 1 replica to Deployment(name='MultiDecoder', app='video-analyzer'). (ServeController pid=251181) INFO 2026-01-06 08:27:23,512 controller 251181 -- Adding 2 replicas to Deployment(name='VideoAnalyzer', app='video-analyzer'). (ProxyActor pid=251250) INFO 2026-01-06 08:27:23,416 proxy 10.0.45.10 -- Started <ray.serve._private.router.SharedRouterLongPollClient object at 0x72d0e5f4f710>. (ProxyActor pid=251250) INFO 2026-01-06 08:27:27,576 proxy 10.0.45.10 -- Got updated endpoints: {Deployment(name='VideoAnalyzer', app='video-analyzer'): EndpointInfo(route='/', app_is_cross_language=False, route_patterns=[RoutePattern(methods=['POST'], path='/analyze'), RoutePattern(methods=['GET', 'HEAD'], path='/docs'), RoutePattern(methods=['GET', 'HEAD'], path='/docs/oauth2-redirect'), RoutePattern(methods=['GET'], path='/health'), RoutePattern(methods=['GET', 'HEAD'], path='/openapi.json'), RoutePattern(methods=['GET', 'HEAD'], path='/redoc')])}. (ProxyActor pid=67913, ip=10.0.239.104) INFO 2026-01-06 08:27:28,373 proxy 10.0.239.104 -- Proxy starting on node 6954457300d89edbf779f01ef0bfcfc5d109d927fca0e3b4f80974ba (HTTP port: 8000). [repeated 2x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.) (ProxyActor pid=6627, ip=10.0.222.50) INFO 2026-01-06 08:27:26,287 proxy 10.0.222.50 -- Got updated endpoints: {Deployment(name='VideoAnalyzer', app='video-analyzer'): EndpointInfo(route='/', app_is_cross_language=False, route_patterns=None)}.
(ServeReplica:video-analyzer:VideoEncoder pid=6736, ip=10.0.222.50) VideoEncoder initializing on cuda
(ServeReplica:video-analyzer:VideoEncoder pid=6736, ip=10.0.222.50) Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`. (ProxyActor pid=67913, ip=10.0.239.104) INFO 2026-01-06 08:27:28,444 proxy 10.0.239.104 -- Started <ray.serve._private.router.SharedRouterLongPollClient object at 0x7dd178a4d190>. [repeated 2x across cluster]
(ServeReplica:video-analyzer:VideoEncoder pid=6736, ip=10.0.222.50) VideoEncoder ready (embedding_dim=1152)
INFO 2026-01-06 08:27:31,518 serve 250708 -- Application 'video-analyzer' is ready at http://0.0.0.0:8000/. INFO 2026-01-06 08:27:31,534 serve 250708 -- Started <ray.serve._private.router.SharedRouterLongPollClient object at 0x78edac0dbef0>.
DeploymentHandle(deployment='VideoAnalyzer')
```python
import httpx
import time
# Wait for deployment to be ready
print("Waiting for deployment to be healthy...")
start = time.time()
while True:
try:
with httpx.Client(timeout=5.0) as client:
response = client.get("http://localhost:8000/health")
if response.status_code == 200:
print(f"Deployment ready in {time.time() - start:.1f}s")
break
except httpx.RequestError:
pass
time.sleep(1.0)
if time.time() - start > 120:
raise TimeoutError("Deployment did not become healthy within 120s")
```
Waiting for deployment to be healthy... Deployment ready in 0.0s
```python
import httpx
import time
import uuid
# Send a sample request to the deployed service
payload = {
"stream_id": uuid.uuid4().hex[:8],
"video_path": SAMPLE_VIDEO_URI,
"num_frames": 16,
"chunk_duration": 10.0,
}
print(f"Analyzing video: {SAMPLE_VIDEO_URI}")
print(f"Stream ID: {payload['stream_id']}\n")
start = time.perf_counter()
with httpx.Client(timeout=300.0) as client:
response = client.post("http://localhost:8000/analyze", json=payload)
latency_ms = (time.perf_counter() - start) * 1000
result = response.json()
# Print results
print("=" * 60)
print(f"Video duration: {result['video_duration']:.1f}s")
print(f"Chunks processed: {result['num_chunks']}")
print()
print("Top Tags:")
for tag in result["tags"]:
print(f" {tag['score']:.3f} {tag['text']}")
print()
print("Best Caption:")
print(f" {result['retrieval_caption']['score']:.3f} {result['retrieval_caption']['text']}")
print()
print(f"Scene Changes: {result['num_scene_changes']}")
for sc in result["scene_changes"][:5]: # Show first 5
print(f" {sc['timestamp']:6.2f}s score={sc['score']:.3f}")
print()
print("Timing:")
timing = result["timing_ms"]
print(f" S3 download: {timing['s3_download_ms']:7.1f} ms")
print(f" Video decode: {timing['decode_video_ms']:7.1f} ms")
print(f" Encode (GPU): {timing['encode_ms']:7.1f} ms")
print(f" Decode (CPU): {timing['decode_ms']:7.1f} ms")
print(f" Total server: {timing['total_ms']:7.1f} ms")
print(f" Round-trip: {latency_ms:7.1f} ms")
print("=" * 60)
```
(ServeReplica:video-analyzer:VideoAnalyzer pid=67729, ip=10.0.239.104) INFO 2026-01-06 08:27:31,893 video-analyzer_VideoAnalyzer npdr89ay b9cd7535-7acc-4fa4-a54b-a0e789ca9ce7 -- GET /health 200 1.5ms
Analyzing video: s3://anyscale-example-video-analysis-test-bucket/anyscale-example/stock-videos/kitchen_cooking_35395675_00.mp4 Stream ID: 766cc773
(ServeReplica:video-analyzer:VideoAnalyzer pid=67728, ip=10.0.239.104) INFO 2026-01-06 08:27:32,683 video-analyzer_VideoAnalyzer ntukkwkd 9326cc34-5b87-491c-8025-a7db9a2806f4 -- Started <ray.serve._private.router.SharedRouterLongPollClient object at 0x788bd83d2120>.
(ServeReplica:video-analyzer:VideoEncoder pid=6736, ip=10.0.222.50) Unbatched: 16 frames ============================================================ Video duration: 8.3s Chunks processed: 1
Top Tags: 0.082 kitchen 0.070 cafe 0.058 living room 0.057 bedroom 0.053 office
Best Caption: 0.101 Someone preparing food on a counter
Scene Changes: 0
Timing: S3 download: 238.8 ms Video decode: 100.6 ms Encode (GPU): 18.3 ms Decode (CPU): 855.0 ms Total server: 1212.9 ms Round-trip: 1235.5 ms ============================================================
(ServeReplica:video-analyzer:MultiDecoder pid=6737, ip=10.0.222.50) /home/ray/anaconda3/lib/python3.12/site-packages/ray/serve/_private/replica.py:1640: UserWarning: Calling sync method '__call__' directly on the asyncio loop. In a future version, sync methods will be run in a threadpool by default. Ensure your sync methods are thread safe or keep the existing behavior by making them `async def`. Opt into the new behavior by setting RAY_SERVE_RUN_SYNC_IN_THREADPOOL=1. (ServeReplica:video-analyzer:MultiDecoder pid=6737, ip=10.0.222.50) warnings.warn( (ServeReplica:video-analyzer:MultiDecoder pid=6737, ip=10.0.222.50) INFO 2026-01-06 08:27:33,539 video-analyzer_MultiDecoder 1carxuil 9326cc34-5b87-491c-8025-a7db9a2806f4 -- CALL __call__ OK 2.1ms (ServeReplica:video-analyzer:VideoEncoder pid=6736, ip=10.0.222.50) INFO 2026-01-06 08:27:33,533 video-analyzer_VideoEncoder pxjowb4x 9326cc34-5b87-491c-8025-a7db9a2806f4 -- CALL __call__ OK 804.7ms (ServeReplica:video-analyzer:VideoAnalyzer pid=67728, ip=10.0.239.104) INFO 2026-01-06 08:27:33,541 video-analyzer_VideoAnalyzer ntukkwkd 9326cc34-5b87-491c-8025-a7db9a2806f4 -- POST /analyze 200 1217.9ms
```python
serve.shutdown()
```
---
## Load Test Results
To evaluate the pipeline's performance under realistic conditions, we ran load tests using the `client/load_test.py` script with **concurrency levels ranging from 2 to 64 concurrent requests**. The Ray Serve application was configured with autoscaling enabled, allowing replicas to scale dynamically based on demand.
### Methodology
The load test was executed using:
```bash
python -m client.load_test --video s3://bucket/video.mp4 --concurrency <N>
```
**Test parameters:**
- Concurrency levels: 2, 4, 8, 16, 32, 64
- Chunk duration: 10 seconds
- Frames per chunk: 16
- Autoscaling: Enabled (replicas scale based on `target_num_ongoing_requests`)
The charts below show autoscaling in action during the concurrency=2 to 64 load test:
![Ongoing Requests](assets/ongoing.png)
![Replica Count](assets/num_replicas.png)
![Queue Length](assets/queue_len.png)
As load increases, replicas scale up to maintain target queue depth. The system reaches steady state once enough replicas are provisioned to handle the request rate.
To ensure fair comparison, we discarded the first half of each test run to exclude the autoscaling warm-up period where latencies are elevated due to replica initialization.
### Results Summary
| Concurrency | Requests | P50 (ms) | P95 (ms) | P99 (ms) | Throughput (req/s) |
|-------------|----------|----------|----------|----------|-------------------|
| 2 | 152 | 838 | 843 | 844 | 2.37 |
| 4 | 590 | 858 | 984 | 1,025 | 4.68 |
| 8 | 1,103 | 885 | 1,065 | 1,120 | 8.97 |
| 16 | 2,240 | 900 | 1,074 | 1,128 | 17.78 |
| 32 | 4,141 | 928 | 1,095 | 1,151 | 34.75 |
| 64 | 17,283 | 967 | 1,128 | 1,188 | 67.55 |
**Key findings:**
- **100% success rate** across all 25,509 requests analyzed
- **Near-linear throughput scaling**: Throughput increased from 2.37 req/s at concurrency 2 to **67.55 req/s at concurrency 64** (~28x improvement)
- **Stable latencies under load**: P95 latency remained between 843ms and 1,128ms across all concurrency levels, demonstrating effective autoscaling
### Processing Time Breakdown
The chart below shows how processing time is distributed across pipeline stages at each concurrency level:
![Processing Time Breakdown](assets/processing_time_breakdown.png)
**At concurrency 64 (best throughput):**
- **S3 Download**: 77ms (8%)
- **Video Decode (FFmpeg)**: 98ms (10%)
- **Encode (GPU)**: <1ms (<1%) — Note: This number is artificially low due to how timing is measured. Because the output of VideoEncoder is passed directly into MultiDecoder as an argument, the instrumentation mistakenly attributes most of the computational time to the downstream stage. In reality, VideoEncoder performs the bulk of the work, so its true processing time is significantly higher than reported here.
- **Decode (CPU)**: 757ms (81%)
The CPU decoding stage (tag/caption generation) dominates processing time. S3 download and video decoding are relatively fast due to local caching and optimized FFmpeg settings.
### Throughput Analysis
The charts below show throughput scaling and the latency-throughput tradeoff:
![Throughput Analysis](assets/throughput_analysis.png)
**Observations:**
1. **Linear scaling**: Throughput scales almost linearly with concurrency, indicating that autoscaling successfully provisions enough replicas to handle increased load.
2. **Latency-throughput tradeoff**: The right chart shows that P95 latency increases slightly as throughput grows (from ~843ms to ~1,128ms), but remains within acceptable bounds. This ~34% latency increase enables a ~28x throughput improvement.
3. **No saturation point**: Even at concurrency 64, throughput continues to scale. The system could likely handle higher concurrency with additional GPU resources.
The near-linear scaling demonstrates that the pipeline architecture—with separate GPU encoder, CPU decoder, and CPU-bound ingress—allows each component to scale independently based on its resource requirements.
doc/source/serve/tutorials/video-analysis/__init__.py
View File
doc/source/serve/tutorials/video-analysis/analyze_load_tests.ipynb
+1382
View File
File diff suppressed because one or more lines are too long
doc/source/serve/tutorials/video-analysis/app.py
+357
View File
@@ -0,0 +1,357 @@
"""
Ray Serve application: Video Embedding → Multi-Decoder.
Processes entire videos by chunking into segments.
Videos are downloaded from S3 to temp file, then processed locally (faster than streaming).
Encoder refs are passed directly to decoder; Ray Serve resolves dependencies automatically.
Usage:
serve run app:app
# With custom bucket:
S3_BUCKET=my-bucket serve run app:app
"""
import logging
import os
import tempfile
import time
from collections import defaultdict
from pathlib import Path
from urllib.parse import urlparse
import aioboto3
import numpy as np
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from ray import serve
from ray.serve.handle import DeploymentResponse
from deployments.encoder import VideoEncoder
from deployments.decoder import MultiDecoder
from utils.video import chunk_video_async
from constants import DEFAULT_NUM_FRAMES, DEFAULT_CHUNK_DURATION, FFMPEG_THREADS, NUM_WORKERS
logger = logging.getLogger(__name__)
def parse_s3_uri(s3_uri: str) -> tuple[str, str]:
"""Parse s3://bucket/key into (bucket, key)."""
parsed = urlparse(s3_uri)
if parsed.scheme != "s3":
raise ValueError(f"Invalid S3 URI: {s3_uri}")
bucket = parsed.netloc
key = parsed.path.lstrip("/")
return bucket, key
class AnalyzeRequest(BaseModel):
"""Request schema for /analyze endpoint."""
stream_id: str
video_path: str # S3 URI: s3://bucket/key
num_frames: int = DEFAULT_NUM_FRAMES
chunk_duration: float = DEFAULT_CHUNK_DURATION
use_batching: bool = False # Set False to compare unbatched performance
class TagResult(BaseModel):
text: str
score: float
class CaptionResult(BaseModel):
text: str
score: float
class TimingResult(BaseModel):
s3_download_ms: float
decode_video_ms: float
encode_ms: float
decode_ms: float
total_ms: float
class SceneChange(BaseModel):
"""Detected scene change event."""
timestamp: float # Seconds from video start
score: float # Scene change score (higher = bigger change)
chunk_index: int
frame_index: int # Frame index within chunk
class ChunkResult(BaseModel):
"""Result for a single chunk."""
chunk_index: int
start_time: float
duration: float
tags: list[TagResult]
retrieval_caption: CaptionResult
# Detected scene changes in this chunk
scene_changes: list[SceneChange]
class AnalyzeResponse(BaseModel):
"""Response schema for /analyze endpoint."""
stream_id: str
# Aggregated results (across all chunks)
tags: list[TagResult]
retrieval_caption: CaptionResult
# Scene change detection
scene_changes: list[SceneChange] # All detected scene changes
num_scene_changes: int
# Per-chunk results
chunks: list[ChunkResult]
num_chunks: int
video_duration: float
timing_ms: TimingResult
# FastAPI app
fastapi_app = FastAPI(
title="Video Embedding API",
description="GPU encoder → CPU multi-decoder using SigLIP embeddings",
)
@serve.deployment(
# setting this to twice that of the encoder. So that requests can complete the
# upfront CPU work and be queued for GPU processing.
num_replicas="auto",
ray_actor_options={"num_cpus": FFMPEG_THREADS},
max_ongoing_requests=4,
autoscaling_config={
"min_replicas": 2,
"max_replicas": 20,
"target_num_ongoing_requests": 2,
},
)
@serve.ingress(fastapi_app)
class VideoAnalyzer:
"""
Main ingress deployment that orchestrates VideoEncoder and MultiDecoder.
Encoder refs are passed directly to decoder; Ray Serve resolves dependencies.
Downloads video from S3 to temp file for fast local processing.
"""
def __init__(self, encoder: VideoEncoder, decoder: MultiDecoder):
self.encoder = encoder
self.decoder = decoder
self._s3_session = aioboto3.Session()
self._s3_client = None # Cached client for reuse across requests
logger.info("VideoAnalyzer ready")
async def _get_s3_client(self):
"""Get or create a reusable S3 client."""
if self._s3_client is None:
self._s3_client = await self._s3_session.client("s3").__aenter__()
return self._s3_client
async def _download_video(self, s3_uri: str) -> Path:
"""Download video from S3 to temp file. Returns local path."""
bucket, key = parse_s3_uri(s3_uri)
# Create temp file with video extension
suffix = Path(key).suffix or ".mp4"
temp_file = tempfile.NamedTemporaryFile(suffix=suffix, delete=False)
temp_path = Path(temp_file.name)
temp_file.close()
try:
s3 = await self._get_s3_client()
await s3.download_file(bucket, key, str(temp_path))
except Exception:
# Clean up temp file if download fails
temp_path.unlink(missing_ok=True)
raise
return temp_path
def _aggregate_results(
self,
chunk_results: list[dict],
top_k_tags: int = 5,
) -> dict:
"""
Aggregate results from multiple chunks.
Strategy:
- Tags: Average scores across chunks, return top-k
- Caption: Return the one with highest score across all chunks
"""
# Aggregate tag scores
tag_scores = defaultdict(list)
for result in chunk_results:
for tag in result["tags"]:
tag_scores[tag["text"]].append(tag["score"])
# Average tag scores and sort
aggregated_tags = [
{"text": text, "score": np.mean(scores)}
for text, scores in tag_scores.items()
]
aggregated_tags.sort(key=lambda x: x["score"], reverse=True)
top_tags = aggregated_tags[:top_k_tags]
# Best caption across all chunks
best_caption = max(
(r["retrieval_caption"] for r in chunk_results),
key=lambda x: x["score"],
)
return {
"tags": top_tags,
"retrieval_caption": best_caption,
}
def _encode_chunk(self, frames: np.ndarray, use_batching: bool = False) -> DeploymentResponse:
"""Encode a single chunk's frames to embeddings. Returns DeploymentResponse ref."""
return self.encoder.remote(frames, use_batching=use_batching)
async def _decode_chunk(
self,
encoder_output: dict,
chunk_index: int,
chunk_start_time: float,
chunk_duration: float,
ema_state=None,
) -> dict:
"""Decode embeddings to tags, caption, scene changes."""
return await self.decoder.remote(
encoder_output=encoder_output,
chunk_index=chunk_index,
chunk_start_time=chunk_start_time,
chunk_duration=chunk_duration,
ema_state=ema_state,
)
@fastapi_app.post("/analyze", response_model=AnalyzeResponse)
async def analyze(self, request: AnalyzeRequest) -> AnalyzeResponse:
"""
Analyze a video from S3 and return tags, caption, and scene changes.
Downloads video to temp file for fast local processing.
Chunks the entire video and aggregates results.
Encoder refs are passed directly to decoder for dependency resolution.
"""
total_start = time.perf_counter()
temp_path = None
try:
# Download video from S3 to temp file
download_start = time.perf_counter()
try:
temp_path = await self._download_video(request.video_path)
except Exception as e:
raise HTTPException(status_code=400, detail=f"Cannot download S3 video: {e}")
s3_download_ms = (time.perf_counter() - download_start) * 1000
# Chunk video with PARALLEL frame extraction from local file
decode_start = time.perf_counter()
try:
chunks = await chunk_video_async(
str(temp_path),
chunk_duration=request.chunk_duration,
num_frames_per_chunk=request.num_frames,
ffmpeg_threads=FFMPEG_THREADS,
use_single_ffmpeg=True,
)
except Exception as e:
raise HTTPException(status_code=400, detail=f"Cannot process video: {e}")
decode_video_ms = (time.perf_counter() - decode_start) * 1000
if not chunks:
raise HTTPException(status_code=400, detail="No chunks extracted from video")
# Calculate video duration from chunks
video_duration = chunks[-1].start_time + chunks[-1].duration
# Fire off all encoder calls (returns refs, not awaited)
encode_start = time.perf_counter()
encode_refs = [
self._encode_chunk(chunk.frames, use_batching=request.use_batching)
for chunk in chunks
]
encode_ms = (time.perf_counter() - encode_start) * 1000
# Decode chunks SERIALLY, passing encoder refs directly.
# Ray Serve resolves the encoder result when decoder needs it.
# EMA state is tracked here (not in decoder) to ensure continuity
# even when autoscaling routes requests to different replicas.
decode_start = time.perf_counter()
decode_results = []
ema_state = None # Will be initialized from first chunk's first frame
for chunk, enc_ref in zip(chunks, encode_refs):
dec_result = await self._decode_chunk(
encoder_output=enc_ref,
chunk_index=chunk.index,
chunk_start_time=chunk.start_time,
chunk_duration=chunk.duration,
ema_state=ema_state,
)
decode_results.append(dec_result)
ema_state = dec_result["ema_state"] # Carry forward for next chunk
decode_ms = (time.perf_counter() - decode_start) * 1000
# Collect results
chunk_results = []
per_chunk_results = []
all_scene_changes = []
for chunk, decoder_result in zip(chunks, decode_results):
chunk_results.append(decoder_result)
# Scene changes come directly from decoder
chunk_scene_changes = [
SceneChange(**sc) for sc in decoder_result["scene_changes"]
]
all_scene_changes.extend(chunk_scene_changes)
per_chunk_results.append(ChunkResult(
chunk_index=chunk.index,
start_time=chunk.start_time,
duration=chunk.duration,
tags=[TagResult(**t) for t in decoder_result["tags"]],
retrieval_caption=CaptionResult(**decoder_result["retrieval_caption"]),
scene_changes=chunk_scene_changes,
))
# Aggregate results
aggregated = self._aggregate_results(chunk_results)
total_ms = (time.perf_counter() - total_start) * 1000
return AnalyzeResponse(
stream_id=request.stream_id,
tags=[TagResult(**t) for t in aggregated["tags"]],
retrieval_caption=CaptionResult(**aggregated["retrieval_caption"]),
scene_changes=all_scene_changes,
num_scene_changes=len(all_scene_changes),
chunks=per_chunk_results,
num_chunks=len(chunks),
video_duration=video_duration,
timing_ms=TimingResult(
s3_download_ms=round(s3_download_ms, 2),
decode_video_ms=round(decode_video_ms, 2),
encode_ms=round(encode_ms, 2),
decode_ms=round(decode_ms, 2),
total_ms=round(total_ms, 2),
),
)
finally:
# Clean up temp file
if temp_path and temp_path.exists():
temp_path.unlink(missing_ok=True)
@fastapi_app.get("/health")
async def health(self):
"""Health check endpoint."""
return {"status": "healthy"}
encoder = VideoEncoder.bind()
decoder = MultiDecoder.bind(bucket=os.environ.get("S3_BUCKET"))
app = VideoAnalyzer.bind(encoder=encoder, decoder=decoder)
doc/source/serve/tutorials/video-analysis/assets/chunk_video_analysis.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 146 KiB

doc/source/serve/tutorials/video-analysis/assets/chunk_video_heatmap.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 96 KiB

doc/source/serve/tutorials/video-analysis/assets/decord_vs_ffmpeg_local.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 70 KiB

doc/source/serve/tutorials/video-analysis/assets/ffmpeg_threads_benchmark.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 57 KiB

doc/source/serve/tutorials/video-analysis/assets/gpu_utilization.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 121 KiB

doc/source/serve/tutorials/video-analysis/assets/latency_distribution_by_concurrency.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 114 KiB

doc/source/serve/tutorials/video-analysis/assets/latency_histograms.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 166 KiB

doc/source/serve/tutorials/video-analysis/assets/latency_over_time.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 710 KiB

doc/source/serve/tutorials/video-analysis/assets/num_replicas.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 121 KiB

doc/source/serve/tutorials/video-analysis/assets/ongoing.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 111 KiB

doc/source/serve/tutorials/video-analysis/assets/presigned_vs_direct_s3.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 74 KiB

doc/source/serve/tutorials/video-analysis/assets/processing_time_breakdown.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 64 KiB

doc/source/serve/tutorials/video-analysis/assets/processing_time_pie_charts.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 176 KiB

doc/source/serve/tutorials/video-analysis/assets/queue_len.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 109 KiB

doc/source/serve/tutorials/video-analysis/assets/single_vs_multi_ffmpeg.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 62 KiB

doc/source/serve/tutorials/video-analysis/assets/sync_vs_async_comparison.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 52 KiB

doc/source/serve/tutorials/video-analysis/assets/thread_pool_benchmark.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 63 KiB

doc/source/serve/tutorials/video-analysis/assets/throughput_analysis.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 97 KiB

doc/source/serve/tutorials/video-analysis/autoscaling_policy.py
+110
View File
@@ -0,0 +1,110 @@
"""
Application-level autoscaling policy for video processing pipeline.
Scaling strategy:
- VideoAnalyzer: scales based on its load (error_ratio = requests / capacity)
- VideoEncoder: scales based on its load, with floor = VideoAnalyzer replicas
- MultiDecoder: 0.5x VideoEncoder replicas
Error ratio formula:
error_ratio = total_ongoing_requests / (target_per_replica × current_replicas)
- error_ratio > 1.0 → over capacity → scale up
- error_ratio < 1.0 → under capacity → scale down
- error_ratio = 1.0 → at capacity → no change
"""
import math
from typing import Dict, Tuple
from ray.serve._private.common import DeploymentID
from ray.serve.config import AutoscalingContext
def _get_error_ratio(ctx: AutoscalingContext) -> float:
"""
Calculate error ratio: how much over/under target capacity we are.
Returns 1.0 when idle to maintain current replicas.
"""
target_per_replica = ctx.config.target_ongoing_requests or 1
total_requests = ctx.total_num_requests
current_replicas = ctx.current_num_replicas
if total_requests == 0:
return 1.0 # Idle: maintain current replicas
total_capacity = target_per_replica * current_replicas
return total_requests / total_capacity
def _scale_by_error_ratio(ctx: AutoscalingContext, floor: int = 0) -> int:
"""
Calculate target replicas based on error ratio.
Args:
ctx: Deployment autoscaling context
floor: Minimum replicas (in addition to capacity_adjusted_min)
Returns:
Target replica count, clamped to min/max limits
"""
error_ratio = _get_error_ratio(ctx)
# Scale current replicas by error ratio
target = int(math.ceil(ctx.current_num_replicas * error_ratio))
# Apply floor (e.g., encoder should have at least as many as analyzer)
target = max(target, floor)
# Clamp to configured limits
return max(
ctx.capacity_adjusted_min_replicas,
min(ctx.capacity_adjusted_max_replicas, target),
)
def _find_deployment(
contexts: Dict[DeploymentID, AutoscalingContext],
name: str,
) -> Tuple[DeploymentID, AutoscalingContext]:
"""Find deployment by name."""
for dep_id, ctx in contexts.items():
if dep_id.name == name:
return dep_id, ctx
raise KeyError(f"Deployment '{name}' not found")
def coordinated_scaling_policy(
contexts: Dict[DeploymentID, AutoscalingContext],
) -> Tuple[Dict[DeploymentID, int], Dict]:
"""
Coordinated scaling for video processing pipeline.
Scaling rules:
VideoAnalyzer: scale by its own load
VideoEncoder: scale by its own load, floor = analyzer replicas
MultiDecoder: 0.5x encoder replicas
"""
decisions: Dict[DeploymentID, int] = {}
# 1. VideoAnalyzer: scale by load
analyzer_id, analyzer_ctx = _find_deployment(contexts, "VideoAnalyzer")
analyzer_replicas = _scale_by_error_ratio(analyzer_ctx)
decisions[analyzer_id] = analyzer_replicas
# 2. VideoEncoder: scale by load, but at least as many as analyzer
encoder_id, encoder_ctx = _find_deployment(contexts, "VideoEncoder")
encoder_replicas = _scale_by_error_ratio(encoder_ctx, floor=analyzer_replicas)
decisions[encoder_id] = encoder_replicas
# 3. MultiDecoder: 0.5x encoder replicas
decoder_id, decoder_ctx = _find_deployment(contexts, "MultiDecoder")
decoder_replicas = max(1, math.ceil(encoder_replicas / 2))
decoder_replicas = max(
decoder_ctx.capacity_adjusted_min_replicas,
min(decoder_ctx.capacity_adjusted_max_replicas, decoder_replicas),
)
decisions[decoder_id] = decoder_replicas
return decisions, {}
doc/source/serve/tutorials/video-analysis/chunk_video_benchmark.ipynb
+1741
View File
File diff suppressed because one or more lines are too long
doc/source/serve/tutorials/video-analysis/ci/aws.yaml
+19
View File
@@ -0,0 +1,19 @@
cloud_id: {{ env["ANYSCALE_CLOUD_ID"] }}
region: us-west-2
# Cluster resources
head_node_type:
name: head
instance_type: m5.2xlarge
resources:
cpu: 8
worker_node_types:
- name: 16CPU-64GB
instance_type: m8i.4xlarge
min_workers: 1
max_workers: 32
- name: 4CPU-16GB-1GPU
instance_type: g6.xlarge
min_workers: 1
max_workers: 32
doc/source/serve/tutorials/video-analysis/ci/build.sh
Executable
+21
View File
@@ -0,0 +1,21 @@
#!/bin/bash
set -euxo pipefail
# Install Python dependencies
pip3 install --no-cache-dir \
"numpy>=1.24.0,<2.0" \
"torch>=2.0.0" \
"transformers>=4.35.0" \
"accelerate>=0.25.0" \
"sentencepiece>=0.1.99" \
"httpx>=0.25.0" \
"aioboto3>=12.0.0" \
"pillow>=10.0.0"
sudo apt-get update && sudo apt-get install -y --no-install-recommends \
ffmpeg \
&& sudo rm -rf /var/lib/apt/lists/*
export S3_BUCKET=anyscale-example-video-analysis-test-bucket
doc/source/serve/tutorials/video-analysis/ci/gce.yaml
+19
View File
@@ -0,0 +1,19 @@
cloud_id: {{ env["ANYSCALE_CLOUD_ID"] }} # required; set by CI
region: us-central1
# Cluster resources
head_node_type:
name: head
instance_type: n1-standard-8
resources:
cpu: 8
worker_node_types:
- name: 16CPU-64GB-GPU
instance_type: n4-standard-16
min_workers: 1
max_workers: 32
- name: 4CPU-16GB-GPU
instance_type: g2-standard-4
min_workers: 1
max_workers: 32
doc/source/serve/tutorials/video-analysis/ci/nb2py.py
+40
View File
@@ -0,0 +1,40 @@
#!/usr/bin/env python3
import argparse
import nbformat
def convert_notebook(input_path: str, output_path: str) -> None:
"""
Read a Jupyter notebook and write a Python script.
Cells that load or autoreload extensions are ignored.
"""
nb = nbformat.read(input_path, as_version=4)
with open(output_path, "w") as out:
for cell in nb.cells:
if cell.cell_type != "code":
continue
lines = cell.source.splitlines()
# Skip cells that load or autoreload extensions
if any(
l.strip().startswith("%load_ext autoreload")
or l.strip().startswith("%autoreload all")
for l in lines
):
continue
out.write(cell.source.rstrip() + "\n\n")
def main() -> None:
parser = argparse.ArgumentParser(
description="Convert a Jupyter notebook to a Python script."
)
parser.add_argument("input_nb", help="Path to the input .ipynb file")
parser.add_argument("output_py", help="Path for the output .py script")
args = parser.parse_args()
convert_notebook(args.input_nb, args.output_py)
if __name__ == "__main__":
main()
doc/source/serve/tutorials/video-analysis/ci/tests.sh
Executable
+11
View File
@@ -0,0 +1,11 @@
#!/bin/bash
set -euxo pipefail
S3_BUCKET=anyscale-example-video-analysis-test-bucket
export S3_BUCKET
python ci/nb2py.py README.ipynb README.py # convert notebook to py script
python README.py
rm README.py # remove the generated script
doc/source/serve/tutorials/video-analysis/client/__init__.py
View File
doc/source/serve/tutorials/video-analysis/client/load_test.py
+374
View File
@@ -0,0 +1,374 @@
#!/usr/bin/env python3
"""
Load testing script for the Ray Serve video API.
Usage:
python -m client.load_test --video s3://bucket/path/to/video.mp4 --concurrency 4
python -m client.load_test --video s3://bucket/video.mp4 --concurrency 8 --url http://localhost:8000
python -m client.load_test --video s3://bucket/video.mp4 --concurrency 4 --token YOUR_TOKEN
Press Ctrl+C to stop and save results to CSV.
"""
import argparse
import asyncio
import csv
import signal
import sys
import time
import uuid
from dataclasses import dataclass, field, asdict
from datetime import datetime
from pathlib import Path
from typing import Optional
import httpx
@dataclass
class RequestResult:
"""Result of a single request."""
request_id: str
stream_id: str
start_time: float
end_time: float
latency_ms: float
success: bool
status_code: int
error: Optional[str] = None
# Server-side timing (if successful)
s3_download_ms: Optional[float] = None
decode_video_ms: Optional[float] = None
encode_ms: Optional[float] = None
decode_ms: Optional[float] = None
server_total_ms: Optional[float] = None
# Response data
num_chunks: Optional[int] = None
video_duration: Optional[float] = None
num_scene_changes: Optional[int] = None
@dataclass
class LoadTestStats:
"""Aggregated statistics for the load test."""
total_requests: int = 0
successful_requests: int = 0
failed_requests: int = 0
total_latency_ms: float = 0.0
min_latency_ms: float = float('inf')
max_latency_ms: float = 0.0
latencies: list = field(default_factory=list)
def add_result(self, result: RequestResult):
self.total_requests += 1
if result.success:
self.successful_requests += 1
self.total_latency_ms += result.latency_ms
self.min_latency_ms = min(self.min_latency_ms, result.latency_ms)
self.max_latency_ms = max(self.max_latency_ms, result.latency_ms)
self.latencies.append(result.latency_ms)
else:
self.failed_requests += 1
@property
def avg_latency_ms(self) -> float:
if not self.latencies:
return 0.0
return self.total_latency_ms / len(self.latencies)
@property
def p50_latency_ms(self) -> float:
return self._percentile(50)
@property
def p95_latency_ms(self) -> float:
return self._percentile(95)
@property
def p99_latency_ms(self) -> float:
return self._percentile(99)
def _percentile(self, p: float) -> float:
if not self.latencies:
return 0.0
sorted_latencies = sorted(self.latencies)
idx = int(len(sorted_latencies) * p / 100)
idx = min(idx, len(sorted_latencies) - 1)
return sorted_latencies[idx]
class LoadTester:
def __init__(
self,
video_path: str,
url: str,
concurrency: int,
num_frames: int = 16,
chunk_duration: float = 10.0,
timeout: float = 300.0,
token: Optional[str] = None,
):
self.video_path = video_path
self.url = url
self.concurrency = concurrency
self.num_frames = num_frames
self.chunk_duration = chunk_duration
self.timeout = timeout
self.token = token
self.results: list[RequestResult] = []
self.stats = LoadTestStats()
self.running = True
self.start_time: Optional[float] = None
self.request_counter = 0
self._lock = asyncio.Lock()
self._semaphore: asyncio.Semaphore = None
def _build_payload(self) -> dict:
"""Build the request payload."""
stream_id = uuid.uuid4().hex[:8]
return {
"stream_id": stream_id,
"video_path": self.video_path,
"num_frames": self.num_frames,
"chunk_duration": self.chunk_duration,
"use_batching": False,
}
async def _make_request(self, client: httpx.AsyncClient, request_id: str) -> RequestResult:
"""Make a single request and return the result."""
payload = self._build_payload()
stream_id = payload["stream_id"]
start = time.perf_counter()
start_timestamp = time.time()
headers = {}
if self.token:
headers["Authorization"] = f"Bearer {self.token}"
try:
response = await client.post(f"{self.url}/analyze", json=payload, headers=headers)
end = time.perf_counter()
latency_ms = (end - start) * 1000
if response.status_code == 200:
data = response.json()
timing = data.get("timing_ms", {})
return RequestResult(
request_id=request_id,
stream_id=stream_id,
start_time=start_timestamp,
end_time=time.time(),
latency_ms=latency_ms,
success=True,
status_code=response.status_code,
s3_download_ms=timing.get("s3_download_ms"),
decode_video_ms=timing.get("decode_video_ms"),
encode_ms=timing.get("encode_ms"),
decode_ms=timing.get("decode_ms"),
server_total_ms=timing.get("total_ms"),
num_chunks=data.get("num_chunks"),
video_duration=data.get("video_duration"),
num_scene_changes=data.get("num_scene_changes"),
)
else:
return RequestResult(
request_id=request_id,
stream_id=stream_id,
start_time=start_timestamp,
end_time=time.time(),
latency_ms=latency_ms,
success=False,
status_code=response.status_code,
error=response.text[:200],
)
except Exception as e:
end = time.perf_counter()
latency_ms = (end - start) * 1000
return RequestResult(
request_id=request_id,
stream_id=stream_id,
start_time=start_timestamp,
end_time=time.time(),
latency_ms=latency_ms,
success=False,
status_code=0,
error=str(e)[:200],
)
async def _request_task(self, client: httpx.AsyncClient, request_id: str):
"""Execute a single request and record the result. Releases semaphore when done."""
try:
result = await self._make_request(client, request_id)
async with self._lock:
self.results.append(result)
self.stats.add_result(result)
# Print progress
status = "✓" if result.success else "✗"
print(
f"{status} {result.request_id} | "
f"{result.latency_ms:7.1f}ms | "
f"Total: {self.stats.total_requests} "
f"(OK: {self.stats.successful_requests}, Fail: {self.stats.failed_requests})"
)
finally:
self._semaphore.release()
async def run(self):
"""Run the load test until interrupted."""
self.start_time = time.time()
self._semaphore = asyncio.Semaphore(self.concurrency)
pending_tasks: set[asyncio.Task] = set()
print("=" * 70)
print("🚀 Starting Load Test")
print("=" * 70)
print(f" Video: {self.video_path}")
print(f" URL: {self.url}")
print(f" Concurrency: {self.concurrency}")
print(f" Chunk dur: {self.chunk_duration}s")
print(f" Frames/chunk:{self.num_frames}")
print()
print("Press Ctrl+C to stop and save results...")
print("-" * 70)
async with httpx.AsyncClient(timeout=self.timeout) as client:
while self.running:
# Block until a slot is available
await self._semaphore.acquire()
if not self.running:
self._semaphore.release()
break
# Spawn task (it will release semaphore when done)
self.request_counter += 1
request_id = f"req_{self.request_counter:06d}"
task = asyncio.create_task(self._request_task(client, request_id))
pending_tasks.add(task)
task.add_done_callback(pending_tasks.discard)
# Wait for in-flight requests
if pending_tasks:
print(f"\n⏳ Waiting for {len(pending_tasks)} in-flight requests...")
await asyncio.gather(*pending_tasks, return_exceptions=True)
def stop(self):
"""Stop the load test."""
self.running = False
def save_results(self, output_path: str):
"""Save results to CSV."""
if not self.results:
print("No results to save.")
return
fieldnames = [
"request_id", "stream_id", "start_time", "end_time", "latency_ms",
"success", "status_code", "error",
"s3_download_ms", "decode_video_ms", "encode_ms", "decode_ms", "server_total_ms",
"num_chunks", "video_duration", "num_scene_changes"
]
with open(output_path, "w", newline="") as f:
writer = csv.DictWriter(f, fieldnames=fieldnames)
writer.writeheader()
for result in self.results:
writer.writerow(asdict(result))
print(f"📁 Results saved to: {output_path}")
def print_summary(self):
"""Print summary statistics."""
if not self.results:
print("No results to summarize.")
return
duration = time.time() - self.start_time if self.start_time else 0
throughput = self.stats.total_requests / duration if duration > 0 else 0
print()
print("=" * 70)
print("📊 Load Test Summary")
print("=" * 70)
print(f" Duration: {duration:.1f}s")
print(f" Total requests: {self.stats.total_requests}")
print(f" Successful: {self.stats.successful_requests}")
print(f" Failed: {self.stats.failed_requests}")
print(f" Success rate: {self.stats.successful_requests / self.stats.total_requests * 100:.1f}%")
print(f" Throughput: {throughput:.2f} req/s")
print()
print("⏱️ Latency (successful requests):")
if self.stats.latencies:
print(f" Min: {self.stats.min_latency_ms:8.1f} ms")
print(f" Max: {self.stats.max_latency_ms:8.1f} ms")
print(f" Avg: {self.stats.avg_latency_ms:8.1f} ms")
print(f" P50: {self.stats.p50_latency_ms:8.1f} ms")
print(f" P95: {self.stats.p95_latency_ms:8.1f} ms")
print(f" P99: {self.stats.p99_latency_ms:8.1f} ms")
else:
print(" (no successful requests)")
print("=" * 70)
def main():
parser = argparse.ArgumentParser(description="Load test the Ray Serve video API")
parser.add_argument("--video", type=str, required=True, help="S3 URI: s3://bucket/key")
parser.add_argument("--url", type=str, default="http://127.0.0.1:8000", help="Server URL")
parser.add_argument("--concurrency", type=int, default=4, help="Number of concurrent workers")
parser.add_argument("--num-frames", type=int, default=16, help="Frames per chunk")
parser.add_argument("--chunk-duration", type=float, default=10.0, help="Chunk duration in seconds")
parser.add_argument("--timeout", type=float, default=300.0, help="Request timeout in seconds")
parser.add_argument("--output", type=str, default=None, help="Output CSV path (default: load_test_<timestamp>.csv)")
parser.add_argument("--token", type=str, default=None, help="Bearer token for Authorization header")
args = parser.parse_args()
# Generate output path if not provided
if args.output is None:
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
args.output = f"load_test_{timestamp}_{args.concurrency}.csv"
tester = LoadTester(
video_path=args.video,
url=args.url,
concurrency=args.concurrency,
num_frames=args.num_frames,
chunk_duration=args.chunk_duration,
timeout=args.timeout,
token=args.token,
)
# Track interrupt count for force exit
interrupt_count = 0
# Handle Ctrl+C gracefully
def signal_handler(sig, frame):
nonlocal interrupt_count
interrupt_count += 1
if interrupt_count == 1:
print("\n\n🛑 Stopping load test (press Ctrl+C again to force exit)...")
tester.stop()
else:
print("\n\n⚠️ Force exiting...")
tester.print_summary()
tester.save_results(args.output)
sys.exit(1)
signal.signal(signal.SIGINT, signal_handler)
signal.signal(signal.SIGTERM, signal_handler)
try:
asyncio.run(tester.run())
except KeyboardInterrupt:
pass
finally:
tester.print_summary()
tester.save_results(args.output)
if __name__ == "__main__":
main()
doc/source/serve/tutorials/video-analysis/client/send_video.py
+115
View File
@@ -0,0 +1,115 @@
#!/usr/bin/env python3
"""
Client script to send video to the Ray Serve API.
Usage:
python -m client.send_video --video s3://bucket/path/to/video.mp4
python -m client.send_video --video s3://bucket/video.mp4 --chunk-duration 5.0
python -m client.send_video --video s3://bucket/video.mp4 --token YOUR_TOKEN
"""
import argparse
import time
import uuid
import httpx
def main():
parser = argparse.ArgumentParser(description="Send video to Ray Serve API")
parser.add_argument("--video", type=str, required=True, help="S3 URI: s3://bucket/key")
parser.add_argument("--stream-id", type=str, default=None, help="Stream ID (random if not provided)")
parser.add_argument("--num-frames", type=int, default=16, help="Frames per chunk")
parser.add_argument("--chunk-duration", type=float, default=10.0, help="Chunk duration in seconds")
parser.add_argument("--url", type=str, default="http://127.0.0.1:8000", help="Server URL")
parser.add_argument("--token", type=str, default=None, help="Bearer token for Authorization header")
args = parser.parse_args()
# Generate random stream ID if not provided
stream_id = args.stream_id or uuid.uuid4().hex[:8]
payload = {
"stream_id": stream_id,
"video_path": args.video,
"num_frames": args.num_frames,
"chunk_duration": args.chunk_duration,
"use_batching": True
}
print(f"📹 Processing video: {args.video}")
print(f" Stream ID: {stream_id}")
print(f" Chunk duration: {args.chunk_duration}s, Frames/chunk: {args.num_frames}")
print()
start = time.perf_counter()
headers = {}
if args.token:
headers["Authorization"] = f"Bearer {args.token}"
with httpx.Client(timeout=300.0) as client:
response = client.post(f"{args.url}/analyze", json=payload, headers=headers)
latency_ms = (time.perf_counter() - start) * 1000
if response.status_code != 200:
print(f"❌ Error {response.status_code}: {response.text}")
return
result = response.json()
print("=" * 60)
print("✅ Response")
print("=" * 60)
print(f"Stream ID: {result['stream_id']}")
print(f"Video duration: {result['video_duration']:.1f}s")
print(f"Chunks processed: {result['num_chunks']}")
print()
print("🏷️ Top Tags (aggregated):")
for tag in result["tags"]:
print(f" {tag['score']:.3f} {tag['text']}")
print()
print("📝 Best Caption:")
caption = result["retrieval_caption"]
print(f" {caption['score']:.3f} {caption['text']}")
print()
# Scene changes
scene_changes = result["scene_changes"]
print(f"🎬 Scene Changes Detected: {result['num_scene_changes']}")
if scene_changes:
for sc in scene_changes:
print(f" {sc['timestamp']:6.2f}s score={sc['score']:.3f} (chunk {sc['chunk_index']}, frame {sc['frame_index']})")
else:
print(" (none detected)")
print()
# Show per-chunk results
print("📊 Per-Chunk Results:")
print("-" * 60)
for chunk in result["chunks"]:
print(f" Chunk {chunk['chunk_index']}: {chunk['start_time']:.1f}s - {chunk['start_time'] + chunk['duration']:.1f}s")
print(f" Top tag: {chunk['tags'][0]['text']} ({chunk['tags'][0]['score']:.3f})")
print(f" Caption: {chunk['retrieval_caption']['text'][:50]}...")
num_changes = len(chunk["scene_changes"])
print(f" Scene changes: {num_changes}")
print()
timing = result["timing_ms"]
print("⏱️ Timing:")
print(f" S3 download: {timing['s3_download_ms']:.1f} ms")
print(f" Video decode: {timing['decode_video_ms']:.1f} ms")
print(f" Encode (GPU): {timing['encode_ms']:.1f} ms")
print(f" Decode (CPU): {timing['decode_ms']:.1f} ms")
print(f" Total server: {timing['total_ms']:.1f} ms")
print(f" Round-trip: {latency_ms:.1f} ms")
if result['num_chunks'] > 1:
avg_per_chunk = timing['total_ms'] / result['num_chunks']
print(f" Avg/chunk: {avg_per_chunk:.1f} ms")
if __name__ == "__main__":
main()
doc/source/serve/tutorials/video-analysis/config.yaml
+91
View File
@@ -0,0 +1,91 @@
# This file was generated using the `serve build` command on Ray v2.53.0.
proxy_location: EveryNode
http_options:
host: 0.0.0.0
port: 8000
grpc_options:
port: 9000
grpc_servicer_functions: []
logging_config:
encoding: JSON
log_level: INFO
logs_dir: null
enable_access_log: true
additional_log_standard_attrs: []
applications:
- name: app1
route_prefix: /
import_path: app:app
runtime_env: {}
autoscaling_policy:
policy_function: autoscaling_policy:coordinated_scaling_policy
deployments:
- name: VideoEncoder
num_replicas: "auto"
max_ongoing_requests: 2
autoscaling_config:
min_replicas: 1
initial_replicas: null
max_replicas: 32
target_ongoing_requests: 2.0
metrics_interval_s: 10.0
look_back_period_s: 30.0
smoothing_factor: 1.0
upscale_smoothing_factor: null
downscale_smoothing_factor: null
upscaling_factor: null
downscaling_factor: null
downscale_delay_s: 600.0
downscale_to_zero_delay_s: null
upscale_delay_s: 30.0
aggregation_function: mean
ray_actor_options:
num_cpus: 2.0
num_gpus: 1.0
- name: MultiDecoder
num_replicas: "auto"
max_ongoing_requests: 4
autoscaling_config:
min_replicas: 1
initial_replicas: null
max_replicas: 32
target_ongoing_requests: 2.0
metrics_interval_s: 10.0
look_back_period_s: 30.0
smoothing_factor: 1.0
upscale_smoothing_factor: null
downscale_smoothing_factor: null
upscaling_factor: null
downscaling_factor: null
downscale_delay_s: 600.0
downscale_to_zero_delay_s: null
upscale_delay_s: 30.0
aggregation_function: mean
ray_actor_options:
num_cpus: 1.0
- name: VideoAnalyzer
num_replicas: "auto"
max_ongoing_requests: 4
autoscaling_config:
min_replicas: 1
initial_replicas: null
max_replicas: 64
target_ongoing_requests: 2.0
metrics_interval_s: 10.0
look_back_period_s: 30.0
smoothing_factor: 1.0
upscale_smoothing_factor: null
downscale_smoothing_factor: null
upscaling_factor: null
downscaling_factor: null
downscale_delay_s: 600.0
downscale_to_zero_delay_s: null
upscale_delay_s: 30.0
aggregation_function: mean
ray_actor_options:
num_cpus: 6.0
doc/source/serve/tutorials/video-analysis/constants.py
+60
View File
@@ -0,0 +1,60 @@
# Project constants
# SigLIP model
MODEL_NAME = "google/siglip-so400m-patch14-384"
# S3 paths
S3_VIDEOS_PREFIX = "stock-videos/"
S3_EMBEDDINGS_PREFIX = "embeddings/"
# Scene change detection
SCENE_CHANGE_THRESHOLD = 0.15 # EMA score threshold for detecting scene changes
EMA_ALPHA = 0.9 # EMA decay factor (higher = slower adaptation)
# Pexels API
PEXELS_API_BASE = "https://api.pexels.com/videos"
# Concurrency limits
MAX_CONCURRENT_DOWNLOADS = 5
MAX_CONCURRENT_UPLOADS = 5
# Video normalization defaults (384x384 matches model input size for fastest inference)
NORMALIZE_WIDTH = 384
NORMALIZE_HEIGHT = 384
NORMALIZE_FPS = 30
# Video search queries (for downloading stock videos)
SEARCH_QUERIES = [
"kitchen cooking",
"office meeting",
"street city traffic",
"living room home",
"restaurant cafe",
"parking lot cars",
"classroom students",
"warehouse industrial",
"grocery store shopping",
"gym exercise workout",
"person speaking",
"crowd people walking",
"laptop computer work",
"outdoor nature",
"presentation business",
"conversation talking",
"running jogging",
"dining food",
"shopping mall",
"park outdoor",
]
# Video chunking defaults
DEFAULT_NUM_FRAMES = 16
DEFAULT_CHUNK_DURATION = 10.0
# FFmpeg configuration, restricting to 2 threads to avoid over subscription.
FFMPEG_THREADS = 6
# Setting this to 2 because i am assuming average
# video length in my corpus is 20 seconds.
# 20/10(default chunk duration) = 2
# this means we want to set num_cpus to 4 for deployment.
NUM_WORKERS = 3
doc/source/serve/tutorials/video-analysis/deployments/__init__.py
View File
doc/source/serve/tutorials/video-analysis/deployments/decoder.py
+205
View File
@@ -0,0 +1,205 @@
"""MultiDecoder deployment - CPU-based classification, retrieval, and scene detection."""
import io
import logging
import os
import aioboto3
import numpy as np
from ray import serve
from constants import (
S3_EMBEDDINGS_PREFIX,
SCENE_CHANGE_THRESHOLD,
EMA_ALPHA,
)
from utils.s3 import get_s3_region
logger = logging.getLogger(__name__)
@serve.deployment(
num_replicas="auto",
ray_actor_options={"num_cpus": 1},
max_ongoing_requests=4, # can be set higher than 4, but since the encoder is limited to 4, we need to keep it at 4.
autoscaling_config={
"min_replicas": 1,
"max_replicas": 10,
"target_num_ongoing_requests": 2,
},
)
class MultiDecoder:
"""
Decodes video embeddings into tags, captions, and scene changes.
Uses precomputed text embeddings loaded from S3.
This deployment is stateless - EMA state for scene detection is passed
in and returned with each call, allowing the caller to maintain state
continuity across multiple replicas.
"""
async def __init__(self, bucket: str, s3_prefix: str = S3_EMBEDDINGS_PREFIX):
"""Initialize decoder with text embeddings from S3."""
self.bucket = bucket
self.ema_alpha = EMA_ALPHA
self.scene_threshold = SCENE_CHANGE_THRESHOLD
self.s3_prefix = s3_prefix
logger.info(f"MultiDecoder initializing (bucket={self.bucket}, ema_alpha={self.ema_alpha}, threshold={self.scene_threshold})")
await self._load_embeddings()
logger.info(f"MultiDecoder ready (tags={len(self.tag_texts)}, descriptions={len(self.desc_texts)})")
async def _load_embeddings(self):
"""Load precomputed text embeddings from S3."""
session = aioboto3.Session(region_name=get_s3_region(self.bucket))
async with session.client("s3") as s3:
# Load tag embeddings
tag_key = f"{self.s3_prefix}tag_embeddings.npz"
response = await s3.get_object(Bucket=self.bucket, Key=tag_key)
tag_data = await response["Body"].read()
tag_npz = np.load(io.BytesIO(tag_data), allow_pickle=True)
self.tag_embeddings = tag_npz["embeddings"]
self.tag_texts = tag_npz["texts"].tolist()
# Load description embeddings
desc_key = f"{self.s3_prefix}description_embeddings.npz"
response = await s3.get_object(Bucket=self.bucket, Key=desc_key)
desc_data = await response["Body"].read()
desc_npz = np.load(io.BytesIO(desc_data), allow_pickle=True)
self.desc_embeddings = desc_npz["embeddings"]
self.desc_texts = desc_npz["texts"].tolist()
def _cosine_similarity(self, embedding: np.ndarray, bank: np.ndarray) -> np.ndarray:
"""Compute cosine similarity between embedding and all vectors in bank."""
return bank @ embedding
def _get_top_tags(self, embedding: np.ndarray, top_k: int = 5) -> list[dict]:
"""Get top-k matching tags with scores."""
scores = self._cosine_similarity(embedding, self.tag_embeddings)
top_indices = np.argsort(scores)[::-1][:top_k]
return [
{"text": self.tag_texts[i], "score": float(scores[i])}
for i in top_indices
]
def _get_retrieval_caption(self, embedding: np.ndarray) -> dict:
"""Get best matching description."""
scores = self._cosine_similarity(embedding, self.desc_embeddings)
best_idx = np.argmax(scores)
return {
"text": self.desc_texts[best_idx],
"score": float(scores[best_idx]),
}
def _detect_scene_changes(
self,
frame_embeddings: np.ndarray,
chunk_index: int,
chunk_start_time: float,
chunk_duration: float,
ema_state: np.ndarray | None = None,
) -> tuple[list[dict], np.ndarray]:
"""
Detect scene changes using EMA-based scoring.
score_t = 1 - cosine(E_t, ema_t)
ema_t = α * ema_{t-1} + (1-α) * E_t
Args:
frame_embeddings: (T, D) normalized embeddings
chunk_index: Index of this chunk in the video
chunk_start_time: Start time of chunk in video (seconds)
chunk_duration: Duration of chunk (seconds)
ema_state: EMA state from previous chunk, or None for first chunk
Returns:
Tuple of (scene_changes list, updated ema_state)
"""
num_frames = len(frame_embeddings)
if num_frames == 0:
# Return empty changes and unchanged state (or zeros if no state)
return [], ema_state if ema_state is not None else np.zeros(0)
# Initialize EMA from first frame if no prior state
ema = ema_state.copy() if ema_state is not None else frame_embeddings[0].copy()
scene_changes = []
for frame_idx, embedding in enumerate(frame_embeddings):
# Compute score: how different is current frame from recent history
similarity = float(np.dot(embedding, ema))
score = max(0.0, 1.0 - similarity)
# Detect scene change if score exceeds threshold
if score >= self.scene_threshold:
# Calculate timestamp within video
frame_offset = (frame_idx / max(1, num_frames - 1)) * chunk_duration
timestamp = chunk_start_time + frame_offset
scene_changes.append({
"timestamp": round(timestamp, 3),
"score": round(score, 4),
"chunk_index": chunk_index,
"frame_index": frame_idx,
})
# Update EMA
ema = self.ema_alpha * ema + (1 - self.ema_alpha) * embedding
# Re-normalize
ema = ema / np.linalg.norm(ema)
return scene_changes, ema
def __call__(
self,
encoder_output: dict,
chunk_index: int,
chunk_start_time: float,
chunk_duration: float,
top_k_tags: int = 5,
ema_state: np.ndarray | None = None,
) -> dict:
"""
Decode embeddings into tags, caption, and scene changes.
Args:
encoder_output: Dict with 'frame_embeddings' and 'embedding_dim'
chunk_index: Index of this chunk in the video
chunk_start_time: Start time of chunk (seconds)
chunk_duration: Duration of chunk (seconds)
top_k_tags: Number of top tags to return
ema_state: EMA state from previous chunk for scene detection continuity.
Pass None for the first chunk of a stream.
Returns:
Dict containing tags, retrieval_caption, scene_changes, and updated ema_state.
The caller should pass the returned ema_state to the next chunk's call.
"""
# Get frame embeddings from encoder output
frame_embeddings = encoder_output["frame_embeddings"]
# Calculate pooled embedding (mean across frames, normalized)
pooled_embedding = frame_embeddings.mean(axis=0)
pooled_embedding = pooled_embedding / np.linalg.norm(pooled_embedding)
# Classification and retrieval on pooled embedding
tags = self._get_top_tags(pooled_embedding, top_k=top_k_tags)
caption = self._get_retrieval_caption(pooled_embedding)
# Scene change detection on frame embeddings
scene_changes, new_ema_state = self._detect_scene_changes(
frame_embeddings=frame_embeddings,
chunk_index=chunk_index,
chunk_start_time=chunk_start_time,
chunk_duration=chunk_duration,
ema_state=ema_state,
)
return {
"tags": tags,
"retrieval_caption": caption,
"scene_changes": scene_changes,
"ema_state": new_ema_state,
}
doc/source/serve/tutorials/video-analysis/deployments/encoder.py
+140
View File
@@ -0,0 +1,140 @@
"""VideoEncoder deployment - GPU-based frame encoding using SigLIP."""
import asyncio
import logging
from typing import List
import numpy as np
import torch
from ray import serve
from transformers import AutoModel, AutoProcessor
from constants import MODEL_NAME
from utils.video import frames_to_pil_list
logger = logging.getLogger(__name__)
@serve.deployment(
num_replicas="auto",
ray_actor_options={"num_gpus": 1, "num_cpus": 2},
# GPU utilization is at 100% when this is set to 2. with L4
# aka number on ongoing chunks that can be processed at once.
max_ongoing_requests=2,
autoscaling_config={
"min_replicas": 1,
"max_replicas": 10,
"target_num_ongoing_requests": 2,
},
)
class VideoEncoder:
"""
Encodes video frames into embeddings using SigLIP.
Returns both per-frame embeddings and pooled embedding.
"""
def __init__(self):
self.device = "cuda" if torch.cuda.is_available() else "cpu"
print(f"VideoEncoder initializing on {self.device}")
# Load SigLIP model and processor
self.processor = AutoProcessor.from_pretrained(MODEL_NAME)
self.model = AutoModel.from_pretrained(MODEL_NAME).to(self.device)
self.model.eval()
# Get embedding dimension
self.embedding_dim = self.model.config.vision_config.hidden_size
print(f"VideoEncoder ready (embedding_dim={self.embedding_dim})")
def encode_frames(self, frames: np.ndarray) -> np.ndarray:
"""
Encode frames and return per-frame embeddings.
Args:
frames: np.ndarray of shape (T, H, W, 3) uint8 RGB
Returns:
np.ndarray of shape (T, D) float32, L2-normalized per-frame embeddings
"""
# Convert to PIL images
pil_images = frames_to_pil_list(frames)
# Process images
inputs = self.processor(images=pil_images, return_tensors="pt").to(self.device)
# Get embeddings
with torch.no_grad():
with torch.amp.autocast(device_type=self.device, enabled=self.device == "cuda"):
outputs = self.model.get_image_features(**inputs)
# get_image_features returns BaseModelOutputWithPooling; use pooler_output for embeddings
frame_embeddings = torch.nn.functional.normalize(outputs.pooler_output, p=2, dim=1)
# Move to CPU and convert to numpy
result = frame_embeddings.cpu().numpy().astype(np.float32)
return result
async def encode_unbatched(self, frames: np.ndarray) -> dict:
"""
Unbatched entry point - processes single request directly.
Args:
frames: np.ndarray of shape (T, H, W, 3)
Returns:
dict with 'frame_embeddings' and 'embedding_dim'
"""
print(f"Unbatched: {frames.shape[0]} frames")
frame_embeddings = await asyncio.to_thread(self.encode_frames, frames)
return {
"frame_embeddings": frame_embeddings,
"embedding_dim": self.embedding_dim,
}
@serve.batch(max_batch_size=2, batch_wait_timeout_s=0.1)
async def encode_batched(self, frames_batch: List[np.ndarray]) -> List[dict]:
"""
Batched entry point - collects multiple requests into single GPU call.
Args:
frames_batch: List of frame arrays, each of shape (T, H, W, 3)
Returns:
List of dicts, each with 'frame_embeddings' and 'embedding_dim'
"""
frame_counts = [f.shape[0] for f in frames_batch]
total_frames = sum(frame_counts)
print(f"Batched: {len(frames_batch)} requests ({total_frames} total frames)")
# Concatenate all frames into single batch
all_frames = np.concatenate(frames_batch, axis=0)
# Single forward pass for all frames
all_embeddings = await asyncio.to_thread(self.encode_frames, all_frames)
# Split results back per request
results = []
offset = 0
for n_frames in frame_counts:
chunk_embeddings = all_embeddings[offset:offset + n_frames]
results.append({
"frame_embeddings": chunk_embeddings,
"embedding_dim": self.embedding_dim,
})
offset += n_frames
return results
async def __call__(self, frames: np.ndarray, use_batching: bool = False) -> dict:
"""
Main entry point. Set use_batching=False for direct comparison.
"""
if use_batching:
return await self.encode_batched(frames)
else:
return await self.encode_unbatched(frames)
doc/source/serve/tutorials/video-analysis/jobs/__init__.py
View File
doc/source/serve/tutorials/video-analysis/jobs/generate_text_embeddings.py
+195
View File
@@ -0,0 +1,195 @@
#!/usr/bin/env python3
"""
Ray job to generate SigLIP text embeddings for tags and descriptions.
Usage:
ray job submit --working-dir . -- python jobs/generate_text_embeddings.py --bucket my-bucket
"""
import argparse
import asyncio
import io
import json
import time
import aioboto3
import numpy as np
import ray
import torch
from transformers import AutoModel, AutoProcessor
from constants import MODEL_NAME, S3_EMBEDDINGS_PREFIX
from textbanks import TAGS, DESCRIPTIONS
from utils.s3 import get_s3_region
def load_textbanks() -> tuple[list[str], list[str]]:
"""Load tags and descriptions from textbanks module."""
return TAGS, DESCRIPTIONS
def compute_text_embeddings(
texts: list[str],
processor,
model,
device: str,
batch_size: int = 32,
) -> np.ndarray:
"""Compute normalized text embeddings using SigLIP."""
all_embeddings = []
for i in range(0, len(texts), batch_size):
batch_texts = texts[i:i + batch_size]
# Process text
inputs = processor(
text=batch_texts,
padding="max_length",
truncation=True,
return_tensors="pt",
)
inputs = {k: v.to(device) for k, v in inputs.items()}
# Get embeddings
with torch.no_grad():
outputs = model.get_text_features(**inputs)
# Handle case where outputs is a model output object vs raw tensor
if hasattr(outputs, 'pooler_output'):
embeddings = outputs.pooler_output.cpu().numpy()
else:
embeddings = outputs.cpu().numpy()
embeddings = embeddings / np.linalg.norm(embeddings, axis=1, keepdims=True)
all_embeddings.append(embeddings)
return np.vstack(all_embeddings).astype(np.float32)
async def save_to_s3(
session: aioboto3.Session,
embeddings: np.ndarray,
texts: list[str],
bucket: str,
key: str,
) -> str:
"""Save embeddings and texts to S3 as npz file."""
# Save as npz (embeddings + texts)
buffer = io.BytesIO()
np.savez_compressed(
buffer,
embeddings=embeddings,
texts=np.array(texts, dtype=object),
)
buffer.seek(0)
async with session.client("s3") as s3:
await s3.put_object(
Bucket=bucket,
Key=key,
Body=buffer.getvalue(),
ContentType="application/octet-stream",
)
return f"s3://{bucket}/{key}"
async def generate_and_upload(
bucket: str,
s3_prefix: str = S3_EMBEDDINGS_PREFIX,
) -> dict:
"""Generate embeddings and upload to S3."""
print("=" * 60)
print("Starting text embedding generation")
print("=" * 60)
# Load textbanks
print("\n📚 Loading text banks...")
tags, descriptions = load_textbanks()
print(f" Tags: {len(tags)}")
print(f" Descriptions: {len(descriptions)}")
# Load model
print(f"\n🤖 Loading SigLIP model: {MODEL_NAME}")
device = "cuda" if torch.cuda.is_available() else "cpu"
print(f" Device: {device}")
start = time.time()
processor = AutoProcessor.from_pretrained(MODEL_NAME)
model = AutoModel.from_pretrained(MODEL_NAME).to(device)
model.eval()
load_time = time.time() - start
print(f" Model loaded in {load_time:.1f}s")
# Generate tag embeddings
print("\n🏷️ Generating tag embeddings...")
start = time.time()
tag_embeddings = compute_text_embeddings(tags, processor, model, device)
tag_time = time.time() - start
print(f" Shape: {tag_embeddings.shape}")
print(f" Time: {tag_time:.2f}s")
# Generate description embeddings
print("\n📝 Generating description embeddings...")
start = time.time()
desc_embeddings = compute_text_embeddings(descriptions, processor, model, device)
desc_time = time.time() - start
print(f" Shape: {desc_embeddings.shape}")
print(f" Time: {desc_time:.2f}s")
# Upload to S3 concurrently
print(f"\n☁️ Uploading to S3 bucket: {bucket}")
session = aioboto3.Session(region_name=get_s3_region(bucket))
tag_uri, desc_uri = await asyncio.gather(
save_to_s3(session, tag_embeddings, tags, bucket, f"{s3_prefix}tag_embeddings.npz"),
save_to_s3(session, desc_embeddings, descriptions, bucket, f"{s3_prefix}description_embeddings.npz"),
)
print(f" Tags: {tag_uri}")
print(f" Descriptions: {desc_uri}")
print("\n✅ Done!")
return {
"tag_embeddings": {
"s3_uri": tag_uri,
"shape": list(tag_embeddings.shape),
"count": len(tags),
},
"description_embeddings": {
"s3_uri": desc_uri,
"shape": list(desc_embeddings.shape),
"count": len(descriptions),
},
"model": MODEL_NAME,
"timing": {
"model_load_s": load_time,
"tag_embed_s": tag_time,
"desc_embed_s": desc_time,
},
}
@ray.remote(num_gpus=1)
def generate_embeddings_task(bucket: str, s3_prefix: str = S3_EMBEDDINGS_PREFIX) -> dict:
"""Ray task wrapper for async embedding generation."""
return asyncio.run(generate_and_upload(bucket, s3_prefix))
def main():
parser = argparse.ArgumentParser(description="Generate text embeddings for decoder")
parser.add_argument("--bucket", type=str, required=True, help="S3 bucket name")
args = parser.parse_args()
# Run the task on existing Ray cluster
result = ray.get(generate_embeddings_task.remote(args.bucket))
print("\n" + "=" * 60)
print("Results:")
print("=" * 60)
print(json.dumps(result, indent=2))
if __name__ == "__main__":
main()
doc/source/serve/tutorials/video-analysis/scripts/__init__.py
View File
doc/source/serve/tutorials/video-analysis/scripts/download_stock_videos.py
+635
View File
@@ -0,0 +1,635 @@
#!/usr/bin/env python3
"""
Download stock videos from Pexels and upload to S3 (async version).
Videos are normalized to consistent specs before upload for predictable performance.
Usage:
python scripts/download_stock_videos.py --api-key YOUR_KEY --bucket YOUR_BUCKET
"""
import argparse
import asyncio
import json
import os
import shutil
import subprocess
import sys
import tempfile
from pathlib import Path
from typing import Optional
import aioboto3
import httpx
from botocore.exceptions import ClientError
from constants import (
SEARCH_QUERIES,
PEXELS_API_BASE,
MAX_CONCURRENT_DOWNLOADS,
MAX_CONCURRENT_UPLOADS,
S3_VIDEOS_PREFIX,
NORMALIZE_WIDTH,
NORMALIZE_HEIGHT,
NORMALIZE_FPS,
)
from utils.s3 import get_s3_region
def normalize_video(
input_path: Path,
output_path: Path,
width: int = NORMALIZE_WIDTH,
height: int = NORMALIZE_HEIGHT,
fps: int = NORMALIZE_FPS,
preset: str = "fast",
) -> bool:
"""
Normalize video to consistent specs using ffmpeg.
Applies:
- Resolution scaling with letterboxing to preserve aspect ratio
- Consistent FPS
- H.264 codec with main profile
- 1-second GOP for fast seeking
- Removes audio
Args:
input_path: Path to input video
output_path: Path for normalized output
width: Target width (default 1280)
height: Target height (default 720)
fps: Target FPS (default 30)
preset: x264 encoding preset (default "fast")
Returns:
True if successful, False otherwise
"""
cmd = [
"ffmpeg", "-y",
"-i", str(input_path),
"-vf", f"scale={width}:{height}:force_original_aspect_ratio=decrease,pad={width}:{height}:(ow-iw)/2:(oh-ih)/2,fps={fps}",
"-c:v", "libx264",
"-pix_fmt", "yuv420p",
"-profile:v", "main",
"-preset", preset,
"-g", str(fps), # GOP size = 1 second
"-keyint_min", str(fps),
"-sc_threshold", "0",
"-movflags", "+faststart",
"-an", # Remove audio
str(output_path),
]
result = subprocess.run(cmd, capture_output=True, text=True)
if result.returncode != 0:
print(f" ⚠️ ffmpeg normalization failed: {result.stderr[:200]}")
return False
return True
async def search_videos(
client: httpx.AsyncClient,
api_key: str,
query: str,
per_page: int = 5
) -> list[dict]:
"""Search for videos on Pexels."""
headers = {"Authorization": api_key}
params = {
"query": query,
"per_page": per_page,
"orientation": "landscape",
"size": "medium",
}
try:
response = await client.get(
f"{PEXELS_API_BASE}/search",
headers=headers,
params=params,
timeout=30.0
)
response.raise_for_status()
data = response.json()
return data.get("videos", [])
except httpx.HTTPError as e:
print(f" ⚠️ Error searching for '{query}': {e}")
return []
def get_best_video_file(video: dict, max_width: int = 1280) -> Optional[dict]:
"""Get the best quality video file under max_width."""
video_files = video.get("video_files", [])
# Filter to reasonable sizes and sort by quality
suitable = [
vf for vf in video_files
if vf.get("width", 0) <= max_width and vf.get("quality") in ("hd", "sd")
]
if not suitable:
suitable = video_files
if not suitable:
return None
# Sort by width descending (prefer higher quality)
suitable.sort(key=lambda x: x.get("width", 0), reverse=True)
return suitable[0]
async def download_video(
client: httpx.AsyncClient,
url: str,
dest_path: Path
) -> bool:
"""Download a video file."""
try:
async with client.stream("GET", url, timeout=120.0) as response:
response.raise_for_status()
with open(dest_path, "wb") as f:
async for chunk in response.aiter_bytes(chunk_size=8192):
f.write(chunk)
return True
except Exception as e:
print(f" ⚠️ Download error: {e}")
return False
def sanitize_metadata(metadata: dict) -> dict:
"""Sanitize metadata values to ASCII-only for S3."""
result = {}
for k, v in metadata.items():
# Convert to string and encode as ASCII, replacing non-ASCII chars
val = str(v).encode("ascii", errors="replace").decode("ascii")
result[k] = val
return result
async def upload_to_s3(
s3_client,
local_path: Path,
bucket: str,
s3_key: str,
metadata: dict
) -> bool:
"""Upload a file to S3 with metadata."""
try:
extra_args = {
"ContentType": "video/mp4",
"Metadata": sanitize_metadata(metadata)
}
await s3_client.upload_file(str(local_path), bucket, s3_key, ExtraArgs=extra_args)
return True
except ClientError as e:
print(f" ⚠️ S3 upload error: {e}")
return False
def generate_filename(video: dict, query: str, index: int) -> str:
"""Generate a descriptive filename for the video."""
clean_query = query.replace(" ", "_").replace("/", "-")[:30]
video_id = video.get("id", "unknown")
return f"{clean_query}_{video_id}_{index:02d}.mp4"
async def process_video(
http_client: httpx.AsyncClient,
s3_client,
video: dict,
index: int,
temp_dir: Path,
local_dir: Optional[Path],
bucket: Optional[str],
s3_prefix: str,
download_sem: asyncio.Semaphore,
upload_sem: asyncio.Semaphore,
normalize: bool = True,
normalize_width: int = NORMALIZE_WIDTH,
normalize_height: int = NORMALIZE_HEIGHT,
normalize_fps: int = NORMALIZE_FPS,
) -> Optional[dict]:
"""Download, normalize, and upload a single video."""
video_file = get_best_video_file(video)
if not video_file:
print(f" {index+1:2d}. ⚠️ No suitable video file found, skipping")
return None
filename = generate_filename(video, video["_query"], index)
raw_path = temp_dir / f"raw_{filename}"
normalized_path = temp_dir / filename
print(f" {index+1:2d}. Downloading: {filename}")
download_url = video_file.get("link")
if not download_url:
print(f" ⚠️ No download URL, skipping")
return None
# Download with semaphore
async with download_sem:
if not await download_video(http_client, download_url, raw_path):
return None
raw_size_mb = raw_path.stat().st_size / (1024 * 1024)
print(f" ✅ Downloaded ({raw_size_mb:.1f} MB)")
# Normalize video
if normalize:
print(f" 🔄 Normalizing to {normalize_width}x{normalize_height}@{normalize_fps}fps...")
# Run normalization in thread pool to not block event loop
success = await asyncio.to_thread(
normalize_video,
raw_path,
normalized_path,
normalize_width,
normalize_height,
normalize_fps,
)
if not success:
print(f" ⚠️ Normalization failed, using original")
shutil.move(raw_path, normalized_path)
else:
normalized_size_mb = normalized_path.stat().st_size / (1024 * 1024)
print(f" ✅ Normalized ({raw_size_mb:.1f} MB → {normalized_size_mb:.1f} MB)")
raw_path.unlink(missing_ok=True)
# Update dimensions to normalized values
final_width = normalize_width
final_height = normalize_height
else:
# No normalization, just rename
shutil.move(raw_path, normalized_path)
final_width = video_file.get("width")
final_height = video_file.get("height")
# Copy to local dir if specified
if local_dir:
local_path = local_dir / filename
shutil.copy2(normalized_path, local_path)
print(f" 📁 Saved locally: {local_path}")
result = None
# Upload to S3
if s3_client and bucket:
s3_key = f"{s3_prefix}{filename}"
metadata = {
"pexels_id": str(video.get("id", "")),
"query": video["_query"],
"width": str(final_width),
"height": str(final_height),
"duration": str(video.get("duration", "")),
"photographer": video.get("user", {}).get("name", ""),
"normalized": str(normalize),
}
async with upload_sem:
if await upload_to_s3(s3_client, normalized_path, bucket, s3_key, metadata):
print(f" ☁️ Uploaded to: s3://{bucket}/{s3_key}")
result = {
"filename": filename,
"s3_uri": f"s3://{bucket}/{s3_key}",
"s3_key": s3_key,
"pexels_id": video.get("id"),
"query": video["_query"],
"duration": video.get("duration"),
"width": final_width,
"height": final_height,
}
elif local_dir:
# Local only mode
result = {
"filename": filename,
"local_path": str(local_dir / filename),
"pexels_id": video.get("id"),
"query": video["_query"],
"duration": video.get("duration"),
"width": final_width,
"height": final_height,
}
# Clean up temp file
normalized_path.unlink(missing_ok=True)
return result
async def download_sample_videos(
api_key: str | None = None,
bucket: str | None = None,
total: int = 20,
per_query: int = 1,
local_dir: str | None = None,
dry_run: bool = False,
skip_s3: bool = False,
normalize: bool = True,
width: int = NORMALIZE_WIDTH,
height: int = NORMALIZE_HEIGHT,
fps: int = NORMALIZE_FPS,
s3_prefix: str = S3_VIDEOS_PREFIX,
overwrite: bool = True,
) -> list[str]:
"""Download videos from Pexels, normalize them, and upload to S3.
If a manifest already exists in S3, returns the existing video paths
without downloading new videos.
Args:
api_key: Pexels API key. Falls back to PEXELS_API_KEY env var.
bucket: S3 bucket name. Falls back to S3_BUCKET env var.
total: Total number of videos to download.
per_query: Number of videos per search query.
local_dir: Optional local directory to save videos.
dry_run: If True, only show what would be downloaded.
skip_s3: If True, skip S3 upload (local download only).
normalize: If True, normalize videos to consistent specs.
width: Normalized video width.
height: Normalized video height.
fps: Normalized video FPS.
Returns:
List of video paths (S3 URIs or local paths).
"""
# Get configuration from args or environment
api_key = api_key or os.environ.get("PEXELS_API_KEY")
bucket = bucket or os.environ.get("S3_BUCKET")
if not bucket and not skip_s3:
print("❌ Error: S3 bucket required (--bucket or S3_BUCKET)")
print(" Set it or use --skip-s3 to only download locally")
sys.exit(1)
# Setup S3 session
session = aioboto3.Session(region_name=get_s3_region(bucket))
s3_client = None
if not skip_s3:
async with session.client("s3") as s3:
try:
await s3.head_bucket(Bucket=bucket)
print(f"✅ S3 bucket '{bucket}' accessible")
except ClientError as e:
print(f"❌ Error: Cannot access S3 bucket '{bucket}': {e}")
sys.exit(1)
if overwrite:
print("🔄 Overwriting existing manifest")
await s3.delete_object(Bucket=bucket, Key=f"{s3_prefix}manifest.json")
# Check if manifest already exists - return early if so
manifest_key = f"{s3_prefix}manifest.json"
try:
response = await s3.get_object(Bucket=bucket, Key=manifest_key)
manifest_data = await response["Body"].read()
manifest = json.loads(manifest_data.decode("utf-8"))
# Extract paths from existing manifest
video_paths = []
for v in manifest.get("videos", []):
if "s3_uri" in v:
video_paths.append(v["s3_uri"])
elif "local_path" in v:
video_paths.append(v["local_path"])
print(f"✅ Found existing manifest with {len(video_paths)} videos in S3")
print(f" Skipping Pexels API download")
return video_paths
except ClientError as e:
if e.response["Error"]["Code"] != "NoSuchKey":
raise
# Manifest doesn't exist, continue with download
print("📥 No existing manifest found, will download from Pexels")
# Need API key for downloading
if not api_key:
print("❌ Error: Pexels API key required (--api-key or PEXELS_API_KEY)")
print(" Get your free API key at: https://www.pexels.com/api/")
sys.exit(1)
# Create local directory if specified
local_dir_path = None
if local_dir:
local_dir_path = Path(local_dir)
local_dir_path.mkdir(parents=True, exist_ok=True)
print(f"📁 Local directory: {local_dir_path}")
# Create temp directory for downloads
temp_dir = Path(tempfile.mkdtemp(prefix="pexels_videos_"))
print(f"📁 Temp directory: {temp_dir}")
# Track downloaded videos
video_ids_seen = set()
print(f"\n🔍 Searching for {total} videos across {len(SEARCH_QUERIES)} queries...\n")
# Search and collect videos concurrently
all_videos = []
async with httpx.AsyncClient() as http_client:
# Search all queries concurrently
search_tasks = [
search_videos(http_client, api_key, query, per_page=per_query + 2)
for query in SEARCH_QUERIES
]
results = await asyncio.gather(*search_tasks)
for query, videos in zip(SEARCH_QUERIES, results):
print(f" Found {len(videos)} for '{query}'")
for video in videos:
if len(all_videos) >= total:
break
video_id = video.get("id")
if video_id and video_id not in video_ids_seen:
video_ids_seen.add(video_id)
video["_query"] = query
all_videos.append(video)
all_videos = all_videos[:total]
print(f"\n📹 Selected {len(all_videos)} unique videos\n")
if dry_run:
print("🔍 DRY RUN - Would download these videos:\n")
for i, video in enumerate(all_videos):
video_file = get_best_video_file(video)
if video_file:
filename = generate_filename(video, video["_query"], i)
print(f" {i+1:2d}. {filename}")
print(f" URL: {video_file.get('link', 'N/A')[:80]}...")
print(f" Size: {video_file.get('width')}x{video_file.get('height')}")
print()
return []
# Download and upload videos concurrently
if normalize:
print(f"⬇️ Downloading, normalizing ({width}x{height}@{fps}fps), and uploading videos...\n")
else:
print("⬇️ Downloading and uploading videos (no normalization)...\n")
download_sem = asyncio.Semaphore(MAX_CONCURRENT_DOWNLOADS)
upload_sem = asyncio.Semaphore(MAX_CONCURRENT_UPLOADS)
downloaded_videos = []
async with httpx.AsyncClient() as http_client:
if skip_s3:
# No S3, just download
tasks = [
process_video(
http_client, None, video, i, temp_dir, local_dir_path,
None, s3_prefix, download_sem, upload_sem,
normalize=normalize,
normalize_width=width,
normalize_height=height,
normalize_fps=fps,
)
for i, video in enumerate(all_videos)
]
results = await asyncio.gather(*tasks, return_exceptions=True)
downloaded_videos = [r for r in results if r is not None and not isinstance(r, Exception)]
for r in results:
if isinstance(r, Exception):
print(f" ⚠️ Task failed: {r}")
else:
# With S3
async with session.client("s3") as s3_client:
tasks = [
process_video(
http_client, s3_client, video, i, temp_dir, local_dir_path,
bucket, s3_prefix, download_sem, upload_sem,
normalize=normalize,
normalize_width=width,
normalize_height=height,
normalize_fps=fps,
)
for i, video in enumerate(all_videos)
]
results = await asyncio.gather(*tasks, return_exceptions=True)
downloaded_videos = [r for r in results if r is not None and not isinstance(r, Exception)]
# Log any exceptions
for r in results:
if isinstance(r, Exception):
print(f" ⚠️ Task failed: {r}")
# Save manifest
manifest = {
"total_videos": len(downloaded_videos),
"s3_bucket": bucket if not skip_s3 else None,
"s3_prefix": s3_prefix if not skip_s3 else None,
"local_dir": str(local_dir_path) if local_dir_path else None,
"normalized": normalize,
"normalize_settings": {
"width": width,
"height": height,
"fps": fps,
} if normalize else None,
"videos": downloaded_videos,
}
manifest_path = Path("video_manifest.json")
with open(manifest_path, "w") as f:
json.dump(manifest, f, indent=2)
print(f"\n📋 Manifest saved to: {manifest_path}")
# Also upload manifest to S3
if not skip_s3 and bucket:
async with session.client("s3") as s3_client:
manifest_s3_key = f"{s3_prefix}manifest.json"
try:
await s3_client.put_object(
Bucket=bucket,
Key=manifest_s3_key,
Body=json.dumps(manifest, indent=2),
ContentType="application/json"
)
print(f"☁️ Manifest uploaded to: s3://{bucket}/{manifest_s3_key}")
except ClientError as e:
print(f"⚠️ Failed to upload manifest: {e}")
# Cleanup temp dir
try:
temp_dir.rmdir()
except OSError:
pass # May not be empty if some downloads failed
print(f"\n✅ Done! Processed {len(downloaded_videos)} videos.")
# Extract paths from downloaded videos
video_paths = []
for v in downloaded_videos:
if "s3_uri" in v:
video_paths.append(v["s3_uri"])
elif "local_path" in v:
video_paths.append(v["local_path"])
if video_paths:
print("\n📝 Sample paths for testing:")
for path in video_paths[:5]:
print(f" {path}")
return video_paths
def main():
parser = argparse.ArgumentParser(
description="Download Pexels videos, normalize them, and upload to S3",
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog="""
Normalization applies:
- Resolution scaling with letterboxing (preserves aspect ratio)
- Consistent FPS
- H.264 codec (libx264, main profile)
- 1-second GOP for fast seeking
- Removes audio
Examples:
# Download and normalize to default 1280x720@30fps
python scripts/download_stock_videos.py --api-key KEY --bucket BUCKET
# Custom resolution
python scripts/download_stock_videos.py --api-key KEY --bucket BUCKET --width 1920 --height 1080 --fps 24
# Skip normalization (upload original files)
python scripts/download_stock_videos.py --api-key KEY --bucket BUCKET --no-normalize
"""
)
parser.add_argument("--api-key", type=str, help="Pexels API key (or set PEXELS_API_KEY)")
parser.add_argument("--bucket", type=str, help="S3 bucket name (or set S3_BUCKET)")
parser.add_argument("--total", type=int, default=20, help="Total videos to download")
parser.add_argument("--per-query", type=int, default=1, help="Videos per search query")
parser.add_argument("--local-dir", type=str, help="Also save videos locally to this directory")
parser.add_argument("--dry-run", action="store_true", help="Just show what would be downloaded")
parser.add_argument("--skip-s3", action="store_true", help="Skip S3 upload, only download locally")
# Normalization options
parser.add_argument("--no-normalize", action="store_true",
help="Skip video normalization (upload original files)")
parser.add_argument("--width", type=int, default=NORMALIZE_WIDTH,
help=f"Normalized video width (default: {NORMALIZE_WIDTH})")
parser.add_argument("--height", type=int, default=NORMALIZE_HEIGHT,
help=f"Normalized video height (default: {NORMALIZE_HEIGHT})")
parser.add_argument("--fps", type=int, default=NORMALIZE_FPS,
help=f"Normalized video FPS (default: {NORMALIZE_FPS})")
args = parser.parse_args()
asyncio.run(download_sample_videos(
api_key=args.api_key,
bucket=args.bucket,
total=args.total,
per_query=args.per_query,
local_dir=args.local_dir,
dry_run=args.dry_run,
skip_s3=args.skip_s3,
normalize=not args.no_normalize,
width=args.width,
height=args.height,
fps=args.fps,
))
if __name__ == "__main__":
main()
doc/source/serve/tutorials/video-analysis/services.yaml
+98
View File
@@ -0,0 +1,98 @@
# View the docs https://docs.anyscale.com/reference/service-api#serviceconfig.
name: video-analysis-app
image_uri: # add image uri here
compute_config:
head_node:
instance_type: m5.2xlarge
worker_nodes:
- instance_type: m8i.4xlarge
min_nodes: 1
max_nodes: 32
- instance_type: g6.xlarge
min_nodes: 1
max_nodes: 32
working_dir: .
excludes:
- .git
- .env
- .venv
- '**/*.egg-info/**'
- '**/.DS_Store/**'
- '**/__pycache__/**'
applications:
- name: app1
route_prefix: /
import_path: app:app
runtime_env: {}
autoscaling_policy:
policy_function: autoscaling_policy:coordinated_scaling_policy
deployments:
- name: VideoEncoder
num_replicas: "auto"
max_ongoing_requests: 2
autoscaling_config:
min_replicas: 1
initial_replicas: null
max_replicas: 32
target_ongoing_requests: 2.0
metrics_interval_s: 10.0
look_back_period_s: 30.0
smoothing_factor: 1.0
upscale_smoothing_factor: null
downscale_smoothing_factor: null
upscaling_factor: null
downscaling_factor: null
downscale_delay_s: 600.0
downscale_to_zero_delay_s: null
upscale_delay_s: 30.0
aggregation_function: mean
ray_actor_options:
num_cpus: 2.0
num_gpus: 1.0
- name: MultiDecoder
num_replicas: "auto"
max_ongoing_requests: 4
autoscaling_config:
min_replicas: 1
initial_replicas: null
max_replicas: 32
target_ongoing_requests: 2.0
metrics_interval_s: 10.0
look_back_period_s: 30.0
smoothing_factor: 1.0
upscale_smoothing_factor: null
downscale_smoothing_factor: null
upscaling_factor: null
downscaling_factor: null
downscale_delay_s: 600.0
downscale_to_zero_delay_s: null
upscale_delay_s: 30.0
aggregation_function: mean
ray_actor_options:
num_cpus: 1.0
- name: VideoAnalyzer
num_replicas: "auto"
max_ongoing_requests: 4
autoscaling_config:
min_replicas: 1
initial_replicas: null
max_replicas: 64
target_ongoing_requests: 2.0
metrics_interval_s: 10.0
look_back_period_s: 30.0
smoothing_factor: 1.0
upscale_smoothing_factor: null
downscale_smoothing_factor: null
upscaling_factor: null
downscaling_factor: null
downscale_delay_s: 600.0
downscale_to_zero_delay_s: null
upscale_delay_s: 30.0
aggregation_function: mean
ray_actor_options:
num_cpus: 6.0
doc/source/serve/tutorials/video-analysis/textbanks/__init__.py
+5
View File
@@ -0,0 +1,5 @@
from .tags import TAGS
from .descriptions import DESCRIPTIONS
__all__ = ["TAGS", "DESCRIPTIONS"]
doc/source/serve/tutorials/video-analysis/textbanks/descriptions.py
+45
View File
@@ -0,0 +1,45 @@
# Descriptions for caption retrieval
DESCRIPTIONS = [
"A person cooking in a kitchen",
"Someone preparing food on a counter",
"A chef working in a professional kitchen",
"People eating at a dining table",
"A group having a meal together",
"A person working at a desk",
"Someone typing on a laptop",
"A business meeting in progress",
"A presentation being given",
"People collaborating in an office",
"A teacher lecturing in a classroom",
"Students sitting at desks",
"A person giving a speech",
"Someone writing on a whiteboard",
"A customer shopping in a store",
"People browsing products on shelves",
"A cashier at a checkout counter",
"A person exercising at a gym",
"Someone lifting weights",
"A person running on a treadmill",
"People walking on a city sidewalk",
"Pedestrians crossing a street",
"Traffic moving through an intersection",
"Cars driving on a road",
"A vehicle parked in a lot",
"People walking through a park",
"Someone jogging outdoors",
"A group having a conversation",
"Two people talking face to face",
"A person on a phone call",
"Someone reading a book",
"A person watching television",
"People waiting in line",
"A crowded public space",
"An empty hallway or corridor",
"A person entering a building",
"Someone opening a door",
"A delivery being made",
"A person carrying boxes",
"Workers in a warehouse",
]
doc/source/serve/tutorials/video-analysis/textbanks/tags.py
+24
View File
@@ -0,0 +1,24 @@
# Tags for zero-shot scene classification (short category labels)
TAGS = [
"kitchen",
"living room",
"office",
"meeting room",
"classroom",
"restaurant",
"cafe",
"grocery store",
"gym",
"warehouse",
"parking lot",
"city street",
"park",
"shopping mall",
"beach",
"sports field",
"hallway",
"lobby",
"bathroom",
"bedroom",
]
doc/source/serve/tutorials/video-analysis/utils/__init__.py
View File
doc/source/serve/tutorials/video-analysis/utils/s3.py
+8
View File
@@ -0,0 +1,8 @@
import boto3
def get_s3_region(bucket: str) -> str:
"""Get the region of the S3 bucket"""
s3 = boto3.client("s3")
response = s3.get_bucket_location(Bucket=bucket)
# AWS returns None for us-east-1, otherwise returns the region name
return response["LocationConstraint"] or "us-east-1"
doc/source/serve/tutorials/video-analysis/utils/video.py
+473
View File
@@ -0,0 +1,473 @@
"""Video loading and frame sampling utilities using ffmpeg."""
import asyncio
import json
import subprocess
from dataclasses import dataclass
from typing import Optional
import numpy as np
from PIL import Image
from constants import NUM_WORKERS
@dataclass
class VideoMetadata:
"""Video metadata extracted from ffprobe."""
duration: float # seconds
fps: float
width: int
height: int
num_frames: int
def get_video_metadata(video_path: str) -> VideoMetadata:
"""Get video metadata using ffprobe. Works with local files and URLs."""
# Use JSON output for reliable field parsing (CSV order is unpredictable)
cmd = [
"ffprobe",
"-v", "error",
"-select_streams", "v:0",
"-show_entries", "stream=width,height,r_frame_rate,nb_frames,duration",
"-of", "json",
video_path,
]
result = subprocess.run(cmd, capture_output=True, text=True, check=True)
data = json.loads(result.stdout)
stream = data["streams"][0]
width = int(stream["width"])
height = int(stream["height"])
# Parse frame rate (can be "30/1" or "29.97")
fps_str = stream["r_frame_rate"]
if "/" in fps_str:
num, den = fps_str.split("/")
fps = float(num) / float(den)
else:
fps = float(fps_str)
# nb_frames might be N/A for some formats
try:
num_frames = int(stream.get("nb_frames", 0))
except (ValueError, TypeError):
num_frames = 0
# Duration might be in stream or need to be fetched from format
try:
duration = float(stream.get("duration", 0))
except (ValueError, TypeError):
duration = 0
if duration == 0:
# Fallback: get duration from format
cmd2 = [
"ffprobe",
"-v", "error",
"-show_entries", "format=duration",
"-of", "json",
video_path,
]
result2 = subprocess.run(cmd2, capture_output=True, text=True, check=True)
data2 = json.loads(result2.stdout)
duration = float(data2["format"]["duration"])
if num_frames == 0:
num_frames = int(duration * fps)
return VideoMetadata(
duration=duration,
fps=fps,
width=width,
height=height,
num_frames=num_frames,
)
def extract_frames_ffmpeg(
video_path: str,
start_time: float,
duration: float,
num_frames: int,
target_size: int = 384,
ffmpeg_threads: int = 0,
) -> np.ndarray:
"""
Extract frames from a video segment using ffmpeg.
Works with local files and URLs (including presigned S3 URLs).
Args:
video_path: Path to video file or URL
start_time: Start time in seconds
duration: Duration to extract in seconds
num_frames: Number of frames to extract (uniformly sampled)
target_size: Output frame size (square)
ffmpeg_threads: Number of threads for FFmpeg (0 = auto)
Returns:
np.ndarray of shape (num_frames, target_size, target_size, 3) uint8 RGB
"""
# Calculate output fps to get exactly num_frames
output_fps = num_frames / duration if duration > 0 else num_frames
cmd = [
"ffmpeg",
"-threads", str(ffmpeg_threads),
"-ss", str(start_time),
"-t", str(duration),
"-i", video_path,
"-vf", f"fps={output_fps},scale={target_size}:{target_size}",
"-pix_fmt", "rgb24",
"-f", "rawvideo",
"-",
]
result = subprocess.run(
cmd,
capture_output=True,
check=True,
)
# Parse raw video frames
frame_size = target_size * target_size * 3
raw_data = result.stdout
actual_frames = len(raw_data) // frame_size
if actual_frames == 0:
raise ValueError(f"No frames extracted from {video_path} at {start_time}s")
frames = np.frombuffer(raw_data[:actual_frames * frame_size], dtype=np.uint8)
frames = frames.reshape(actual_frames, target_size, target_size, 3)
# Pad or truncate to exact num_frames
if len(frames) < num_frames:
# Pad by repeating last frame
padding = np.tile(frames[-1:], (num_frames - len(frames), 1, 1, 1))
frames = np.concatenate([frames, padding], axis=0)
elif len(frames) > num_frames:
frames = frames[:num_frames]
return frames
@dataclass
class VideoChunk:
"""Represents a chunk of video to process."""
index: int
start_time: float
duration: float
frames: Optional[np.ndarray] = None
async def extract_frames_async(
video_path: str,
start_time: float,
duration: float,
num_frames: int,
target_size: int = 384,
ffmpeg_threads: int = 0,
) -> np.ndarray:
"""Async wrapper for extract_frames_ffmpeg using thread pool."""
return await asyncio.to_thread(
extract_frames_ffmpeg,
video_path,
start_time,
duration,
num_frames,
target_size,
ffmpeg_threads,
)
def _extract_all_chunks_single_ffmpeg(
video_path: str,
chunk_defs: list[tuple[int, float, float]],
num_frames_per_chunk: int,
target_size: int,
ffmpeg_threads: int = 0,
) -> list[np.ndarray]:
"""
Extract frames for ALL chunks in a single FFmpeg call.
Uses the select filter to pick specific frame timestamps, avoiding
multiple process spawns and file seeks.
Args:
video_path: Path to video file or URL
chunk_defs: List of (index, start_time, duration) tuples
num_frames_per_chunk: Frames to extract per chunk
target_size: Output frame size (square)
ffmpeg_threads: Number of threads for FFmpeg (0 = auto)
Returns:
List of numpy arrays, one per chunk
"""
# Build list of all timestamps to extract
all_timestamps = []
for idx, start, duration in chunk_defs:
# Uniformly sample timestamps within each chunk
for i in range(num_frames_per_chunk):
t = start + (i * duration / num_frames_per_chunk)
all_timestamps.append(t)
if not all_timestamps:
return []
# Build select filter expression: select frames nearest to our timestamps
# Using eq(n,frame_num) would require knowing frame numbers, so instead
# we use pts-based selection with a small tolerance
# The 'select' filter with 'lt(prev_pts,T)*gte(pts,T)' picks first frame >= T
# For efficiency, we'll extract at a high fps and pick specific frames,
# or use the thumbnail filter. But simplest: extract all frames near our
# timestamps using the 'select' filter.
# Build the select expression for all timestamps
# select='eq(n,0)+eq(n,10)+eq(n,20)...' but we need PTS-based selection
# Better approach: use fps filter to get enough frames, then select in numpy
# Calculate total time span and required fps
min_t = min(all_timestamps)
max_t = max(all_timestamps)
total_duration = max_t - min_t + 0.1 # small buffer
# We need at least len(all_timestamps) frames over total_duration
# But we want to be precise, so let's use select filter with expressions
# Build select expression: for each timestamp T, select frame where pts >= T and prev_pts < T
# This is complex. Simpler approach: output frames at specific PTS values.
# Most efficient single-pass approach: use the 'select' filter with timestamp checks
# select='between(t,T1-eps,T1+eps)+between(t,T2-eps,T2+eps)+...'
eps = 0.02 # 20ms tolerance for frame selection
select_parts = [f"between(t,{t-eps},{t+eps})" for t in all_timestamps]
select_expr = "+".join(select_parts)
cmd = [
"ffmpeg",
"-threads", str(ffmpeg_threads),
"-i", video_path,
"-vf", f"select='{select_expr}',scale={target_size}:{target_size}",
"-vsync", "vfr", # Variable frame rate to preserve selected frames
"-pix_fmt", "rgb24",
"-f", "rawvideo",
"-",
]
result = subprocess.run(cmd, capture_output=True, check=True)
# Parse raw video frames
frame_size = target_size * target_size * 3
raw_data = result.stdout
total_frames = len(raw_data) // frame_size
if total_frames == 0:
raise ValueError(f"No frames extracted from {video_path}")
all_frames = np.frombuffer(raw_data[:total_frames * frame_size], dtype=np.uint8)
all_frames = all_frames.reshape(total_frames, target_size, target_size, 3)
# Split into chunks
chunk_frames = []
frame_idx = 0
for idx, start, duration in chunk_defs:
# Take num_frames_per_chunk frames for this chunk
end_idx = min(frame_idx + num_frames_per_chunk, total_frames)
chunk_data = all_frames[frame_idx:end_idx]
# Pad if needed
if len(chunk_data) < num_frames_per_chunk:
if len(chunk_data) == 0:
# No frames for this chunk, create black frames
chunk_data = np.zeros((num_frames_per_chunk, target_size, target_size, 3), dtype=np.uint8)
else:
padding = np.tile(chunk_data[-1:], (num_frames_per_chunk - len(chunk_data), 1, 1, 1))
chunk_data = np.concatenate([chunk_data, padding], axis=0)
chunk_frames.append(chunk_data)
frame_idx = end_idx
return chunk_frames
async def chunk_video_async(
video_path: str,
chunk_duration: float = 10.0,
num_frames_per_chunk: int = 16,
target_size: int = 384,
use_single_ffmpeg: bool = False,
ffmpeg_threads: int = 0,
) -> list[VideoChunk]:
"""
Split video into fixed-duration chunks with frame extraction.
Works with local files and URLs (including presigned S3 URLs).
Args:
video_path: Path to video file or URL
chunk_duration: Duration of each chunk in seconds
num_frames_per_chunk: Frames to extract per chunk
target_size: Frame size
use_single_ffmpeg: If True, extract all chunks in one FFmpeg call (faster).
If False, use parallel FFmpeg calls per chunk.
ffmpeg_threads: Number of threads for FFmpeg decoding (0 = auto)
Returns:
List of VideoChunk with frames loaded
"""
# Get metadata (sync call, fast)
metadata = await asyncio.to_thread(get_video_metadata, video_path)
# Build chunk definitions
chunk_defs = []
start = 0.0
index = 0
while start < metadata.duration:
duration = min(chunk_duration, metadata.duration - start)
# Skip very short final chunks
if duration < 0.5:
break
chunk_defs.append((index, start, duration))
start += chunk_duration
index += 1
if not chunk_defs:
return []
if use_single_ffmpeg:
# Single FFmpeg call - more efficient, especially for URLs
frame_results = await asyncio.to_thread(
_extract_all_chunks_single_ffmpeg,
video_path,
chunk_defs,
num_frames_per_chunk,
target_size,
ffmpeg_threads,
)
else:
# Multiple parallel FFmpeg calls, limited to NUM_WORKERS concurrency
semaphore = asyncio.Semaphore(NUM_WORKERS)
async def extract_with_limit(idx, start, duration):
async with semaphore:
return await extract_frames_async(
video_path,
start_time=start,
duration=duration,
num_frames=num_frames_per_chunk,
target_size=target_size,
ffmpeg_threads=ffmpeg_threads,
)
extraction_tasks = [
extract_with_limit(idx, start, duration)
for idx, start, duration in chunk_defs
]
frame_results = await asyncio.gather(*extraction_tasks)
# Build chunk objects
chunks = [
VideoChunk(
index=idx,
start_time=start,
duration=duration,
frames=frames,
)
for (idx, start, duration), frames in zip(chunk_defs, frame_results)
]
return chunks
def chunk_video(
video_path: str,
chunk_duration: float = 10.0,
num_frames_per_chunk: int = 16,
target_size: int = 384,
use_single_ffmpeg: bool = True,
ffmpeg_threads: int = 0,
) -> list[VideoChunk]:
"""
Split video into fixed-duration chunks.
Args:
video_path: Path to video file or URL
chunk_duration: Duration of each chunk in seconds
num_frames_per_chunk: Frames to extract per chunk
target_size: Frame size
use_single_ffmpeg: If True, extract all chunks in one FFmpeg call (faster).
If False, use sequential FFmpeg calls per chunk.
ffmpeg_threads: Number of threads for FFmpeg decoding (0 = auto)
Returns:
List of VideoChunk with frames loaded
"""
metadata = get_video_metadata(video_path)
# Build chunk definitions
chunk_defs = []
start = 0.0
index = 0
while start < metadata.duration:
duration = min(chunk_duration, metadata.duration - start)
# Skip very short final chunks
if duration < 0.5:
break
chunk_defs.append((index, start, duration))
start += chunk_duration
index += 1
if not chunk_defs:
return []
if use_single_ffmpeg:
# Single FFmpeg call - more efficient, especially for URLs
frame_results = _extract_all_chunks_single_ffmpeg(
video_path,
chunk_defs,
num_frames_per_chunk,
target_size,
ffmpeg_threads,
)
else:
# Sequential FFmpeg calls (original approach)
frame_results = []
for idx, start, duration in chunk_defs:
frames = extract_frames_ffmpeg(
video_path,
start_time=start,
duration=duration,
num_frames=num_frames_per_chunk,
target_size=target_size,
ffmpeg_threads=ffmpeg_threads,
)
frame_results.append(frames)
# Build chunk objects
chunks = [
VideoChunk(
index=idx,
start_time=start,
duration=duration,
frames=frames,
)
for (idx, start, duration), frames in zip(chunk_defs, frame_results)
]
return chunks
def frames_to_pil_list(frames: np.ndarray) -> list[Image.Image]:
"""Convert numpy frames array to list of PIL Images."""
return [Image.fromarray(frame) for frame in frames]