chore: import upstream snapshot with attribution
CPU tests Workflow / Testing (ubuntu-latest, 3.12) (push) Failing after 1s
CPU tests Workflow / Testing (ubuntu-latest, 3.13) (push) Failing after 0s
Mypy Type Check / Type Check (push) Failing after 0s
Docs/Test WorkFlow / Test docs build (push) Failing after 1s
PR Conflict Labeler / labeling (push) Failing after 1s
Dependency resolution / Resolve [tflite] extra — Python 3.12 (push) Failing after 0s
Smoke Tests / try-all-models (ubuntu-latest, 3.10) (push) Failing after 0s
Smoke Tests / try-all-models (ubuntu-latest, 3.13) (push) Failing after 1s
CPU tests Workflow / build-pkg (push) Failing after 1s
CPU tests Workflow / Testing (ubuntu-latest, 3.10) (push) Failing after 0s
CPU tests Workflow / Testing (ubuntu-latest, 3.11) (push) Failing after 0s
Smoke Tests / try-all-models (macos-latest, 3.10) (push) Has been cancelled
Smoke Tests / try-all-models (macos-latest, 3.13) (push) Has been cancelled
Smoke Tests / try-all-models (windows-latest, 3.10) (push) Has been cancelled
Smoke Tests / try-all-models (windows-latest, 3.13) (push) Has been cancelled
CPU tests Workflow / Testing (macos-latest, 3.10) (push) Has been cancelled
CPU tests Workflow / Testing (macos-latest, 3.13) (push) Has been cancelled
CPU tests Workflow / Testing (windows-latest, 3.10) (push) Has been cancelled
CPU tests Workflow / Testing (windows-latest, 3.13) (push) Has been cancelled
CPU tests Workflow / testing-guardian (push) Has been cancelled
GPU tests Workflow / Testing (push) Has been cancelled
CPU tests Workflow / Testing (ubuntu-latest, 3.12) (push) Failing after 1s
CPU tests Workflow / Testing (ubuntu-latest, 3.13) (push) Failing after 0s
Mypy Type Check / Type Check (push) Failing after 0s
Docs/Test WorkFlow / Test docs build (push) Failing after 1s
PR Conflict Labeler / labeling (push) Failing after 1s
Dependency resolution / Resolve [tflite] extra — Python 3.12 (push) Failing after 0s
Smoke Tests / try-all-models (ubuntu-latest, 3.10) (push) Failing after 0s
Smoke Tests / try-all-models (ubuntu-latest, 3.13) (push) Failing after 1s
CPU tests Workflow / build-pkg (push) Failing after 1s
CPU tests Workflow / Testing (ubuntu-latest, 3.10) (push) Failing after 0s
CPU tests Workflow / Testing (ubuntu-latest, 3.11) (push) Failing after 0s
Smoke Tests / try-all-models (macos-latest, 3.10) (push) Has been cancelled
Smoke Tests / try-all-models (macos-latest, 3.13) (push) Has been cancelled
Smoke Tests / try-all-models (windows-latest, 3.10) (push) Has been cancelled
Smoke Tests / try-all-models (windows-latest, 3.13) (push) Has been cancelled
CPU tests Workflow / Testing (macos-latest, 3.10) (push) Has been cancelled
CPU tests Workflow / Testing (macos-latest, 3.13) (push) Has been cancelled
CPU tests Workflow / Testing (windows-latest, 3.10) (push) Has been cancelled
CPU tests Workflow / Testing (windows-latest, 3.13) (push) Has been cancelled
CPU tests Workflow / testing-guardian (push) Has been cancelled
GPU tests Workflow / Testing (push) Has been cancelled
This commit is contained in:
@@ -0,0 +1,185 @@
|
||||
---
|
||||
description: Run RF-DETR object detection on images, video, and streams. Nano to 2XLarge models with 2.3-17.2 ms latency and up to 60.1 AP on COCO.
|
||||
---
|
||||
|
||||
# Run an RF-DETR Object Detection Model
|
||||
|
||||
RF-DETR is a real-time transformer architecture for object detection, built on a DINOv2 vision transformer backbone. The base models are trained on the Microsoft COCO dataset and achieve state-of-the-art accuracy and latency trade-offs.
|
||||
|
||||
## Pre-trained Checkpoints
|
||||
|
||||
RF-DETR offers model sizes from Nano to 2XLarge, allowing trade-offs between accuracy, latency, and parameter count. All latency numbers were measured on an NVIDIA T4 using TensorRT, FP16, and batch size 1. Core models (Nano to Large) are licensed under Apache 2.0. XLarge and 2XLarge (marked with △) are provided by the [`rfdetr_plus`](https://github.com/roboflow/rf-detr-plus) extension (`pip install rfdetr[plus]`) under the Platform Model License 1.0 and require a Roboflow account.
|
||||
|
||||
| Size | RF-DETR package class | Inference package alias | COCO AP<sub>50</sub> | COCO AP<sub>50:95</sub> | Latency (ms) | Params (M) | Resolution | License |
|
||||
| :--: | :-------------------: | :---------------------- | :------------------: | :---------------------: | :----------: | :--------: | :--------: | :--------: |
|
||||
| N | `RFDETRNano` | `rfdetr-nano` | 67.6 | 48.4 | 2.3 | 30.5 | 384x384 | Apache 2.0 |
|
||||
| S | `RFDETRSmall` | `rfdetr-small` | 72.1 | 53.0 | 3.5 | 32.1 | 512x512 | Apache 2.0 |
|
||||
| M | `RFDETRMedium` | `rfdetr-medium` | 73.6 | 54.7 | 4.4 | 33.7 | 576x576 | Apache 2.0 |
|
||||
| L | `RFDETRLarge` | `rfdetr-large` | 75.1 | 56.5 | 6.8 | 33.9 | 704x704 | Apache 2.0 |
|
||||
| XL | `RFDETRXLarge` △ | `rfdetr-xlarge` | 77.4 | 58.6 | 11.5 | 126.4 | 700x700 | PML 1.0 |
|
||||
| 2XL | `RFDETR2XLarge` △ | `rfdetr-2xlarge` | 78.5 | 60.1 | 17.2 | 126.9 | 880x880 | PML 1.0 |
|
||||
|
||||
> △ Requires the `rfdetr_plus` extension: `pip install rfdetr[plus]`
|
||||
|
||||
## Run on an Image
|
||||
|
||||
Perform inference on an image using either the `rfdetr` package or the `inference` package. To use a different model size, select the corresponding class or alias from the table above.
|
||||
|
||||
=== "rfdetr"
|
||||
|
||||
```python
|
||||
import supervision as sv
|
||||
from rfdetr import RFDETRMedium
|
||||
from rfdetr.assets.coco_classes import COCO_CLASSES
|
||||
|
||||
model = RFDETRMedium()
|
||||
|
||||
detections = model.predict("https://media.roboflow.com/dog.jpg", threshold=0.5)
|
||||
|
||||
labels = [f"{COCO_CLASSES[class_id]}" for class_id in detections.class_id]
|
||||
|
||||
annotated_image = sv.BoxAnnotator().annotate(detections.metadata["source_image"], detections)
|
||||
annotated_image = sv.LabelAnnotator().annotate(annotated_image, detections, labels)
|
||||
```
|
||||
|
||||
=== "inference"
|
||||
|
||||
```python
|
||||
import requests
|
||||
import supervision as sv
|
||||
from PIL import Image
|
||||
from inference import get_model
|
||||
|
||||
model = get_model("rfdetr-medium")
|
||||
|
||||
image = Image.open(requests.get("https://media.roboflow.com/dog.jpg", stream=True).raw)
|
||||
predictions = model.infer(image, confidence=0.5)[0]
|
||||
detections = sv.Detections.from_inference(predictions)
|
||||
|
||||
annotated_image = sv.BoxAnnotator().annotate(image, detections)
|
||||
annotated_image = sv.LabelAnnotator().annotate(annotated_image, detections)
|
||||
```
|
||||
|
||||
!!! note "Using COCO classes vs. fine-tuned model classes"
|
||||
|
||||
`COCO_CLASSES` works for COCO-pretrained models (80 COCO classes, indexed 0-79).
|
||||
For fine-tuned models, use `detections.data["class_name"]` instead — it resolves
|
||||
class names from the checkpoint and works for both COCO and custom datasets.
|
||||
|
||||
For memory-constrained inference-only deployments with the `rfdetr` package, optimize the loaded model in place before
|
||||
calling `predict()`. Pass `dtype="float16"` to halve weight memory in addition to clearing the base model reference.
|
||||
This operation is irreversible — to restore the original model, create a new `RFDETR` instance:
|
||||
|
||||
```python
|
||||
model.optimize_for_inference(compile=False, inplace=True, dtype="float16")
|
||||
```
|
||||
|
||||
## Run on video, webcam, or RTSP stream
|
||||
|
||||
These examples use OpenCV for decoding and display. Replace `<SOURCE_VIDEO_PATH>`, `<WEBCAM_INDEX>`, and `<RTSP_STREAM_URL>` with your inputs. `<WEBCAM_INDEX>` is usually `0` for the default camera.
|
||||
|
||||
=== "video"
|
||||
|
||||
```python
|
||||
import cv2
|
||||
import supervision as sv
|
||||
from rfdetr import RFDETRMedium
|
||||
from rfdetr.assets.coco_classes import COCO_CLASSES
|
||||
|
||||
model = RFDETRMedium()
|
||||
|
||||
video_capture = cv2.VideoCapture("<SOURCE_VIDEO_PATH>")
|
||||
if not video_capture.isOpened():
|
||||
raise RuntimeError("Failed to open video source: <SOURCE_VIDEO_PATH>")
|
||||
|
||||
while True:
|
||||
success, frame_bgr = video_capture.read()
|
||||
if not success:
|
||||
break
|
||||
|
||||
frame_rgb = cv2.cvtColor(frame_bgr, cv2.COLOR_BGR2RGB)
|
||||
detections = model.predict(frame_rgb, threshold=0.5)
|
||||
|
||||
labels = [COCO_CLASSES[class_id] for class_id in detections.class_id]
|
||||
|
||||
annotated_frame = sv.BoxAnnotator().annotate(frame_bgr, detections)
|
||||
annotated_frame = sv.LabelAnnotator().annotate(annotated_frame, detections, labels)
|
||||
|
||||
cv2.imshow("RF-DETR Video", annotated_frame)
|
||||
if cv2.waitKey(1) & 0xFF == ord("q"):
|
||||
break
|
||||
|
||||
video_capture.release()
|
||||
cv2.destroyAllWindows()
|
||||
```
|
||||
|
||||
=== "webcam"
|
||||
|
||||
```python
|
||||
import cv2
|
||||
import supervision as sv
|
||||
from rfdetr import RFDETRMedium
|
||||
from rfdetr.assets.coco_classes import COCO_CLASSES
|
||||
|
||||
model = RFDETRMedium()
|
||||
|
||||
WEBCAM_INDEX = 0
|
||||
video_capture = cv2.VideoCapture(WEBCAM_INDEX)
|
||||
if not video_capture.isOpened():
|
||||
raise RuntimeError(f"Failed to open webcam: {WEBCAM_INDEX}")
|
||||
|
||||
while True:
|
||||
success, frame_bgr = video_capture.read()
|
||||
if not success:
|
||||
break
|
||||
|
||||
frame_rgb = cv2.cvtColor(frame_bgr, cv2.COLOR_BGR2RGB)
|
||||
detections = model.predict(frame_rgb, threshold=0.5)
|
||||
|
||||
labels = [COCO_CLASSES[class_id] for class_id in detections.class_id]
|
||||
|
||||
annotated_frame = sv.BoxAnnotator().annotate(frame_bgr, detections)
|
||||
annotated_frame = sv.LabelAnnotator().annotate(annotated_frame, detections, labels)
|
||||
|
||||
cv2.imshow("RF-DETR Webcam", annotated_frame)
|
||||
if cv2.waitKey(1) & 0xFF == ord("q"):
|
||||
break
|
||||
|
||||
video_capture.release()
|
||||
cv2.destroyAllWindows()
|
||||
```
|
||||
|
||||
=== "stream"
|
||||
|
||||
```python
|
||||
import cv2
|
||||
import supervision as sv
|
||||
from rfdetr import RFDETRMedium
|
||||
from rfdetr.assets.coco_classes import COCO_CLASSES
|
||||
|
||||
model = RFDETRMedium()
|
||||
|
||||
video_capture = cv2.VideoCapture("<RTSP_STREAM_URL>")
|
||||
if not video_capture.isOpened():
|
||||
raise RuntimeError("Failed to open RTSP stream: <RTSP_STREAM_URL>")
|
||||
|
||||
while True:
|
||||
success, frame_bgr = video_capture.read()
|
||||
if not success:
|
||||
break
|
||||
|
||||
frame_rgb = cv2.cvtColor(frame_bgr, cv2.COLOR_BGR2RGB)
|
||||
detections = model.predict(frame_rgb, threshold=0.5)
|
||||
|
||||
labels = [COCO_CLASSES[class_id] for class_id in detections.class_id]
|
||||
|
||||
annotated_frame = sv.BoxAnnotator().annotate(frame_bgr, detections)
|
||||
annotated_frame = sv.LabelAnnotator().annotate(annotated_frame, detections, labels)
|
||||
|
||||
cv2.imshow("RF-DETR RTSP", annotated_frame)
|
||||
if cv2.waitKey(1) & 0xFF == ord("q"):
|
||||
break
|
||||
|
||||
video_capture.release()
|
||||
cv2.destroyAllWindows()
|
||||
```
|
||||
@@ -0,0 +1,203 @@
|
||||
---
|
||||
description: Run RF-DETR keypoint detection on images, video, and streams. COCO-pretrained preview model predicts 17 person keypoints with 71.8 AP at 9.7 ms on NVIDIA T4.
|
||||
---
|
||||
|
||||
# Run an RF-DETR Keypoint Model
|
||||
|
||||
RF-DETR Keypoint is a real-time transformer architecture for keypoint detection, built on a DINOv2 vision transformer backbone. The preview model is pretrained on the Microsoft COCO dataset and predicts 17 body keypoints per detected person.
|
||||
|
||||

|
||||
|
||||
!!! note "Preview model"
|
||||
|
||||
`RFDETRKeypointPreview` is an early-access release. Fine-tuning on custom keypoint datasets is the primary intended use case. See [Keypoint Preview Parameters](../train/training-parameters.md#keypoint-preview-parameters) for training configuration. API surface and checkpoint weights may change before the stable release.
|
||||
|
||||
## Pre-trained Checkpoints
|
||||
|
||||
RF-DETR Keypoint outperforms YOLO26-pose X and YOLO11-pose X at comparable latency on MS COCO. Latency measured on NVIDIA T4, TensorRT FP16, batch size 1.
|
||||
|
||||
{ width=560 }
|
||||
|
||||
| Model | RF-DETR package class | COCO AP<sub>50:95</sub> | Latency (ms) | Params (M) | Resolution | License |
|
||||
| :----------------: | :---------------------: | :---------------------: | :----------: | :--------: | :--------: | :--------: |
|
||||
| Keypoint (Preview) | `RFDETRKeypointPreview` | 71.8 | 9.7 | 126.4 | 576x576 | Apache 2.0 |
|
||||
|
||||
> The keypoint model is available in the `rfdetr` package only. It is not yet available via the `inference` package.
|
||||
|
||||
> Benchmark evaluated on COCO val2017 person keypoints (AP<sub>50:95</sub>) with the standard COCO 17-keypoint OKS sigmas; latency on NVIDIA T4, TensorRT FP16, batch size 1.
|
||||
|
||||
## Run on an Image
|
||||
|
||||
Perform inference on an image using the `rfdetr` package. `model.predict()` returns an [`sv.KeyPoints`](https://supervision.roboflow.com/latest/keypoint/core/) object containing skeleton coordinates and per-keypoint confidence scores for each detected person.
|
||||
|
||||
=== "rfdetr"
|
||||
|
||||
```python
|
||||
import cv2
|
||||
import supervision as sv
|
||||
from rfdetr import RFDETRKeypointPreview
|
||||
|
||||
model = RFDETRKeypointPreview()
|
||||
|
||||
image_bgr = cv2.imread("/path/to/image.jpg")
|
||||
image_rgb = cv2.cvtColor(image_bgr, cv2.COLOR_BGR2RGB)
|
||||
key_points = model.predict(image_rgb, threshold=0.5)
|
||||
|
||||
annotated_image = sv.VertexAnnotator().annotate(image_rgb, key_points)
|
||||
```
|
||||
|
||||

|
||||
|
||||
## Understanding the Output
|
||||
|
||||
`model.predict()` returns an `sv.KeyPoints` object. The fields most commonly used downstream:
|
||||
|
||||
| Field | Shape | Description |
|
||||
| --------------------------------- | -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `key_points.xy` | `(N, K, 2)` | Pixel coordinates of each keypoint per detected instance |
|
||||
| `key_points.keypoint_confidence` | `(N, K)` | Per-keypoint findability score; use to filter low-confidence points |
|
||||
| `key_points.detection_confidence` | `(N,)` | Per-instance detection score; this is what `threshold` filters on. For keypoint models it includes the default uncertainty fusion term normalized to `[0, 1)`. |
|
||||
| `key_points.class_id` | `(N,)` | Model label ID for each detection. COCO-pretrained checkpoints use sparse COCO category IDs (1–90). Fine-tuned active-first keypoint checkpoints use normal 0-based class IDs; in the one-class preview setup, `class_id=0` is the foreground class and `class_id=1` is `"__background__"`. Legacy background-first keypoint checkpoints use slot 0 as `"__background__"` and start foreground classes at slot 1. Use `key_points.data["class_name"]` for name resolution rather than indexing your class list by `class_id`. |
|
||||
| `key_points.data["class_name"]` | `(N,)` | Class names resolved from `class_id`; prefer this over indexing a class-name list directly. |
|
||||
| `key_points.data["xyxy"]` | `(N, 4)` | Bounding box for each detected instance in `[x1, y1, x2, y2]` format |
|
||||
| `key_points.data["source_image"]` | list of arrays | Source frame stored once per detection; all N entries are the same array — use `[0]` to access it |
|
||||
|
||||
`K=17` for the pretrained COCO person-keypoint preview checkpoint. Fine-tuned checkpoints use the keypoint count from their dataset schema, so custom keypoint datasets can return any `K` supported by their COCO keypoint annotations.
|
||||
|
||||
Keypoints with `visible=False` are skipped by supervision annotators. To hide low-confidence joints manually, threshold `key_points.keypoint_confidence` and set matching entries to `False` in `key_points.visible`.
|
||||
|
||||
For fine-tuning on a custom keypoint dataset, see [Keypoint preview custom datasets](../train/index.md#keypoint-preview-custom-datasets).
|
||||
|
||||
## Run on video, webcam, or RTSP stream
|
||||
|
||||
These examples use OpenCV for decoding and display. Replace `<SOURCE_VIDEO_PATH>`, `<WEBCAM_INDEX>`, and `<RTSP_STREAM_URL>` with your inputs. `<WEBCAM_INDEX>` is usually `0` for the default camera.
|
||||
|
||||
=== "video"
|
||||
|
||||
```python
|
||||
import cv2
|
||||
import supervision as sv
|
||||
from rfdetr import RFDETRKeypointPreview
|
||||
|
||||
model = RFDETRKeypointPreview()
|
||||
|
||||
video_capture = cv2.VideoCapture("<SOURCE_VIDEO_PATH>")
|
||||
if not video_capture.isOpened():
|
||||
raise RuntimeError("Failed to open video source: <SOURCE_VIDEO_PATH>")
|
||||
|
||||
while True:
|
||||
success, frame_bgr = video_capture.read()
|
||||
if not success:
|
||||
break
|
||||
|
||||
frame_rgb = cv2.cvtColor(frame_bgr, cv2.COLOR_BGR2RGB)
|
||||
key_points = model.predict(frame_rgb, threshold=0.5)
|
||||
|
||||
annotated_frame = sv.VertexAnnotator().annotate(frame_bgr, key_points)
|
||||
|
||||
cv2.imshow("RF-DETR Keypoint Video", annotated_frame)
|
||||
if cv2.waitKey(1) & 0xFF == ord("q"):
|
||||
break
|
||||
|
||||
video_capture.release()
|
||||
cv2.destroyAllWindows()
|
||||
```
|
||||
|
||||
=== "webcam"
|
||||
|
||||
```python
|
||||
import cv2
|
||||
import supervision as sv
|
||||
from rfdetr import RFDETRKeypointPreview
|
||||
|
||||
model = RFDETRKeypointPreview()
|
||||
|
||||
WEBCAM_INDEX = 0 # Change this to the desired webcam index (e.g., 1, 2, ...)
|
||||
video_capture = cv2.VideoCapture(WEBCAM_INDEX)
|
||||
if not video_capture.isOpened():
|
||||
raise RuntimeError(f"Failed to open webcam: {WEBCAM_INDEX}")
|
||||
|
||||
while True:
|
||||
success, frame_bgr = video_capture.read()
|
||||
if not success:
|
||||
break
|
||||
|
||||
frame_rgb = cv2.cvtColor(frame_bgr, cv2.COLOR_BGR2RGB)
|
||||
key_points = model.predict(frame_rgb, threshold=0.5)
|
||||
|
||||
annotated_frame = sv.VertexAnnotator().annotate(frame_bgr, key_points)
|
||||
|
||||
cv2.imshow("RF-DETR Keypoint Webcam", annotated_frame)
|
||||
if cv2.waitKey(1) & 0xFF == ord("q"):
|
||||
break
|
||||
|
||||
video_capture.release()
|
||||
cv2.destroyAllWindows()
|
||||
```
|
||||
|
||||
=== "stream"
|
||||
|
||||
```python
|
||||
import cv2
|
||||
import supervision as sv
|
||||
from rfdetr import RFDETRKeypointPreview
|
||||
|
||||
model = RFDETRKeypointPreview()
|
||||
|
||||
video_capture = cv2.VideoCapture("<RTSP_STREAM_URL>")
|
||||
if not video_capture.isOpened():
|
||||
raise RuntimeError("Failed to open RTSP stream: <RTSP_STREAM_URL>")
|
||||
|
||||
while True:
|
||||
success, frame_bgr = video_capture.read()
|
||||
if not success:
|
||||
break
|
||||
|
||||
frame_rgb = cv2.cvtColor(frame_bgr, cv2.COLOR_BGR2RGB)
|
||||
key_points = model.predict(frame_rgb, threshold=0.5)
|
||||
|
||||
annotated_frame = sv.VertexAnnotator().annotate(frame_bgr, key_points)
|
||||
|
||||
cv2.imshow("RF-DETR Keypoint RTSP", annotated_frame)
|
||||
if cv2.waitKey(1) & 0xFF == ord("q"):
|
||||
break
|
||||
|
||||
video_capture.release()
|
||||
cv2.destroyAllWindows()
|
||||
```
|
||||
|
||||
## Visualization
|
||||
|
||||
`supervision` provides several keypoint annotators. Choose based on what you want to draw.
|
||||
|
||||
=== "EdgeAnnotator"
|
||||
|
||||
Draws skeleton edges (lines between connected joints). Edges where either endpoint has `visible=False` are skipped automatically.
|
||||
|
||||
```python
|
||||
annotated = sv.EdgeAnnotator().annotate(image, key_points)
|
||||
```
|
||||
|
||||
=== "VertexAnnotator"
|
||||
|
||||
Draws a dot at each keypoint. Keypoints with `visible=False` are skipped automatically.
|
||||
|
||||
```python
|
||||
annotated = sv.VertexAnnotator().annotate(image, key_points)
|
||||
```
|
||||
|
||||
=== "VertexEllipseAnnotator"
|
||||
|
||||
Draws covariance ellipses from `key_points.data["covariance"]`, giving a visual footprint of per-keypoint uncertainty.
|
||||
|
||||
```python
|
||||
annotated = sv.VertexEllipseAnnotator().annotate(image, key_points)
|
||||
```
|
||||
|
||||
=== "VertexEllipseHaloAnnotator"
|
||||
|
||||
Draws the same covariance uncertainty with a soft halo for improved contrast on busy backgrounds.
|
||||
|
||||
```python
|
||||
annotated = sv.VertexEllipseHaloAnnotator().annotate(image, key_points)
|
||||
```
|
||||
@@ -0,0 +1,177 @@
|
||||
---
|
||||
description: Run RF-DETR instance segmentation on images, video, and streams. Mask predictions with 3.4-21.8 ms latency using DINOv2 backbone.
|
||||
---
|
||||
|
||||
# Run an RF-DETR Instance Segmentation Model
|
||||
|
||||
RF-DETR is a real-time transformer architecture for instance segmentation, built on a DINOv2 vision transformer backbone. The base models are trained on the Microsoft COCO dataset and achieve strong accuracy and latency trade-offs.
|
||||
|
||||
## Pre-trained Checkpoints
|
||||
|
||||
RF-DETR-Seg offers model sizes from Nano to 2XLarge, allowing trade-offs between accuracy, latency, and parameter count. All latency numbers were measured on an NVIDIA T4 using TensorRT, FP16, and batch size 1.
|
||||
|
||||
| Size | RF-DETR package class | Inference package alias | COCO AP<sub>50</sub> | COCO AP<sub>50:95</sub> | Latency (ms) | Params (M) | Resolution |
|
||||
| :--: | :-------------------: | :---------------------- | :------------------: | :---------------------: | :----------: | :--------: | :--------: |
|
||||
| N | `RFDETRSegNano` | `rfdetr-seg-nano` | 63.0 | 40.3 | 3.4 | 33.6 | 312x312 |
|
||||
| S | `RFDETRSegSmall` | `rfdetr-seg-small` | 66.2 | 43.1 | 4.4 | 33.7 | 384x384 |
|
||||
| M | `RFDETRSegMedium` | `rfdetr-seg-medium` | 68.4 | 45.3 | 5.9 | 35.7 | 432x432 |
|
||||
| L | `RFDETRSegLarge` | `rfdetr-seg-large` | 70.5 | 47.1 | 8.8 | 36.2 | 504x504 |
|
||||
| XL | `RFDETRSegXLarge` | `rfdetr-seg-xlarge` | 72.2 | 48.8 | 13.5 | 38.1 | 624x624 |
|
||||
| 2XL | `RFDETRSeg2XLarge` | `rfdetr-seg-2xlarge` | 73.1 | 49.9 | 21.8 | 38.6 | 768x768 |
|
||||
|
||||
## Run on an Image
|
||||
|
||||
Perform inference on an image using either the `rfdetr` package or the `inference` package. To use a different model size, select the corresponding class or alias from the table above.
|
||||
|
||||
=== "rfdetr"
|
||||
|
||||
```python
|
||||
import supervision as sv
|
||||
from rfdetr import RFDETRSegMedium
|
||||
from rfdetr.assets.coco_classes import COCO_CLASSES
|
||||
|
||||
model = RFDETRSegMedium()
|
||||
|
||||
detections = model.predict("https://media.roboflow.com/dog.jpg", threshold=0.5)
|
||||
|
||||
labels = [f"{COCO_CLASSES[class_id]}" for class_id in detections.class_id]
|
||||
|
||||
annotated_image = sv.MaskAnnotator().annotate(detections.metadata["source_image"], detections)
|
||||
annotated_image = sv.LabelAnnotator().annotate(annotated_image, detections, labels)
|
||||
```
|
||||
|
||||
=== "inference"
|
||||
|
||||
```python
|
||||
import requests
|
||||
import supervision as sv
|
||||
from PIL import Image
|
||||
from inference import get_model
|
||||
|
||||
model = get_model("rfdetr-seg-medium")
|
||||
|
||||
image = Image.open(requests.get("https://media.roboflow.com/dog.jpg", stream=True).raw)
|
||||
predictions = model.infer(image, confidence=0.5)[0]
|
||||
detections = sv.Detections.from_inference(predictions)
|
||||
|
||||
annotated_image = sv.MaskAnnotator().annotate(image, detections)
|
||||
annotated_image = sv.LabelAnnotator().annotate(annotated_image, detections)
|
||||
```
|
||||
|
||||
For memory-constrained inference-only deployments with the `rfdetr` package, optimize the loaded model in place before
|
||||
calling `predict()`. Pass `dtype="float16"` to halve weight memory in addition to clearing the base model reference.
|
||||
This operation is irreversible — to restore the original model, create a new `RFDETR` instance:
|
||||
|
||||
```python
|
||||
model.optimize_for_inference(compile=False, inplace=True, dtype="float16")
|
||||
```
|
||||
|
||||
## Run on video, webcam, or RTSP stream
|
||||
|
||||
These examples use OpenCV for decoding and display. Replace `<SOURCE_VIDEO_PATH>`, `<WEBCAM_INDEX>`, and `<RTSP_STREAM_URL>` with your inputs. `<WEBCAM_INDEX>` is usually `0` for the default camera.
|
||||
|
||||
=== "video"
|
||||
|
||||
```python
|
||||
import cv2
|
||||
import supervision as sv
|
||||
from rfdetr import RFDETRSegMedium
|
||||
from rfdetr.assets.coco_classes import COCO_CLASSES
|
||||
|
||||
model = RFDETRSegMedium()
|
||||
|
||||
video_capture = cv2.VideoCapture("<SOURCE_VIDEO_PATH>")
|
||||
if not video_capture.isOpened():
|
||||
raise RuntimeError("Failed to open video source: <SOURCE_VIDEO_PATH>")
|
||||
|
||||
while True:
|
||||
success, frame_bgr = video_capture.read()
|
||||
if not success:
|
||||
break
|
||||
|
||||
frame_rgb = cv2.cvtColor(frame_bgr, cv2.COLOR_BGR2RGB)
|
||||
detections = model.predict(frame_rgb, threshold=0.5)
|
||||
|
||||
labels = [COCO_CLASSES[class_id] for class_id in detections.class_id]
|
||||
|
||||
annotated_frame = sv.MaskAnnotator().annotate(frame_bgr, detections)
|
||||
annotated_frame = sv.LabelAnnotator().annotate(annotated_frame, detections, labels)
|
||||
|
||||
cv2.imshow("RF-DETR-Seg Video", annotated_frame)
|
||||
if cv2.waitKey(1) & 0xFF == ord("q"):
|
||||
break
|
||||
|
||||
video_capture.release()
|
||||
cv2.destroyAllWindows()
|
||||
```
|
||||
|
||||
=== "webcam"
|
||||
|
||||
```python
|
||||
import cv2
|
||||
import supervision as sv
|
||||
from rfdetr import RFDETRSegMedium
|
||||
from rfdetr.assets.coco_classes import COCO_CLASSES
|
||||
|
||||
model = RFDETRSegMedium()
|
||||
|
||||
WEBCAM_INDEX = 0 # Change this to the desired webcam index (e.g., 1, 2, ...)
|
||||
video_capture = cv2.VideoCapture(WEBCAM_INDEX)
|
||||
if not video_capture.isOpened():
|
||||
raise RuntimeError(f"Failed to open webcam: {WEBCAM_INDEX}")
|
||||
|
||||
while True:
|
||||
success, frame_bgr = video_capture.read()
|
||||
if not success:
|
||||
break
|
||||
|
||||
frame_rgb = cv2.cvtColor(frame_bgr, cv2.COLOR_BGR2RGB)
|
||||
detections = model.predict(frame_rgb, threshold=0.5)
|
||||
|
||||
labels = [COCO_CLASSES[class_id] for class_id in detections.class_id]
|
||||
|
||||
annotated_frame = sv.MaskAnnotator().annotate(frame_bgr, detections)
|
||||
annotated_frame = sv.LabelAnnotator().annotate(annotated_frame, detections, labels)
|
||||
|
||||
cv2.imshow("RF-DETR-Seg Webcam", annotated_frame)
|
||||
if cv2.waitKey(1) & 0xFF == ord("q"):
|
||||
break
|
||||
|
||||
video_capture.release()
|
||||
cv2.destroyAllWindows()
|
||||
```
|
||||
|
||||
=== "stream"
|
||||
|
||||
```python
|
||||
import cv2
|
||||
import supervision as sv
|
||||
from rfdetr import RFDETRSegMedium
|
||||
from rfdetr.assets.coco_classes import COCO_CLASSES
|
||||
|
||||
model = RFDETRSegMedium()
|
||||
|
||||
video_capture = cv2.VideoCapture("<RTSP_STREAM_URL>")
|
||||
if not video_capture.isOpened():
|
||||
raise RuntimeError("Failed to open RTSP stream: <RTSP_STREAM_URL>")
|
||||
|
||||
while True:
|
||||
success, frame_bgr = video_capture.read()
|
||||
if not success:
|
||||
break
|
||||
|
||||
frame_rgb = cv2.cvtColor(frame_bgr, cv2.COLOR_BGR2RGB)
|
||||
detections = model.predict(frame_rgb, threshold=0.5)
|
||||
|
||||
labels = [COCO_CLASSES[class_id] for class_id in detections.class_id]
|
||||
|
||||
annotated_frame = sv.MaskAnnotator().annotate(frame_bgr, detections)
|
||||
annotated_frame = sv.LabelAnnotator().annotate(annotated_frame, detections, labels)
|
||||
|
||||
cv2.imshow("RF-DETR-Seg RTSP", annotated_frame)
|
||||
if cv2.waitKey(1) & 0xFF == ord("q"):
|
||||
break
|
||||
|
||||
video_capture.release()
|
||||
cv2.destroyAllWindows()
|
||||
```
|
||||
Reference in New Issue
Block a user