wehub-resource-sync
105 lines
3.8 KiB
Markdown
105 lines
3.8 KiB
Markdown
<!--[metadata]
|
|
title = "RGBD"
|
|
tags = ["2D", "3D", "Depth", "NYUD", "Pinhole camera"]
|
|
thumbnail = "https://static.rerun.io/rgbd/2fde3a620adc8bd9a5680260f0792d16ac5498bd/480w.png"
|
|
thumbnail_dimensions = [480, 480]
|
|
channel = "release"
|
|
include_in_manifest = true
|
|
build_args = ["--frames=300"]
|
|
-->
|
|
|
|
Visualizes an example recording from [the NYUD dataset](https://cs.nyu.edu/~fergus/datasets/nyu_depth_v2.html) with RGB and Depth channels.
|
|
|
|
<picture data-inline-viewer="examples/rgbd">
|
|
<source media="(max-width: 480px)" srcset="https://static.rerun.io/rgbd/4109d29ed52fa0a8f980fcdd0e9673360c76879f/480w.png">
|
|
<source media="(max-width: 768px)" srcset="https://static.rerun.io/rgbd/4109d29ed52fa0a8f980fcdd0e9673360c76879f/768w.png">
|
|
<source media="(max-width: 1024px)" srcset="https://static.rerun.io/rgbd/4109d29ed52fa0a8f980fcdd0e9673360c76879f/1024w.png">
|
|
<source media="(max-width: 1200px)" srcset="https://static.rerun.io/rgbd/4109d29ed52fa0a8f980fcdd0e9673360c76879f/1200w.png">
|
|
<img src="https://static.rerun.io/rgbd/4109d29ed52fa0a8f980fcdd0e9673360c76879f/full.png" alt="RGBD example screenshot">
|
|
</picture>
|
|
|
|
## Used Rerun types
|
|
|
|
[`Image`](https://www.rerun.io/docs/reference/types/archetypes/image), [`Pinhole`](https://www.rerun.io/docs/reference/types/archetypes/pinhole), [`DepthImage`](https://www.rerun.io/docs/reference/types/archetypes/depth_image)
|
|
|
|
## Background
|
|
|
|
The dataset, known as the NYU Depth V2 dataset, consists of synchronized pairs of RGB and depth frames recorded by the Microsoft Kinect in various indoor scenes.
|
|
This example visualizes one scene of this dataset, and offers a rich source of data for object recognition, scene understanding, depth estimation, and more.
|
|
|
|
## Logging and visualizing with Rerun
|
|
|
|
The visualizations in this example were created with the following Rerun code:
|
|
|
|
### Timelines
|
|
|
|
All data logged using Rerun in the following sections is connected to a specific time.
|
|
Rerun assigns a timestamp to each piece of logged data, and these timestamps are associated with a [`timeline`](https://www.rerun.io/docs/concepts/logging-and-ingestion/timelines).
|
|
|
|
```python
|
|
rr.set_time("time", timestamp=time.timestamp())
|
|
```
|
|
|
|
### Image
|
|
|
|
The example image is logged as [`Image`](https://www.rerun.io/docs/reference/types/archetypes/image) to the `world/camera/image/rgb` entity.
|
|
|
|
```python
|
|
rr.log("world/camera/image/rgb", rr.Image(img_rgb).compress(jpeg_quality=95))
|
|
```
|
|
|
|
### Depth image
|
|
|
|
Pinhole camera is utilized for achieving a 3D view and camera perspective through the use of the [`Pinhole`](https://www.rerun.io/docs/reference/types/archetypes/pinhole).
|
|
|
|
```python
|
|
rr.log(
|
|
"world/camera/image",
|
|
rr.Pinhole(
|
|
resolution=[img_depth.shape[1], img_depth.shape[0]],
|
|
focal_length=0.7 * img_depth.shape[1],
|
|
),
|
|
)
|
|
```
|
|
|
|
Then, the depth image is logged as a [`DepthImage`](https://www.rerun.io/docs/reference/types/archetypes/depth_image) to the `world/camera/image/depth` entity.
|
|
|
|
```python
|
|
rr.log("world/camera/image/depth", rr.DepthImage(img_depth, meter=DEPTH_IMAGE_SCALING))
|
|
```
|
|
|
|
## Run the code
|
|
|
|
To run this example, make sure you have the Rerun repository checked out and the latest SDK installed:
|
|
|
|
```bash
|
|
pip install --upgrade rerun-sdk # install the latest Rerun SDK
|
|
git clone git@github.com:rerun-io/rerun.git # Clone the repository
|
|
cd rerun
|
|
git checkout latest # Check out the commit matching the latest SDK release
|
|
```
|
|
|
|
Install the necessary libraries specified in the requirements file:
|
|
|
|
```bash
|
|
pip install -e examples/python/rgbd
|
|
```
|
|
|
|
To experiment with the provided example, simply execute the main Python script:
|
|
|
|
```bash
|
|
python -m rgbd # run the example
|
|
```
|
|
|
|
You can specify the recording:
|
|
|
|
```bash
|
|
python -m rgbd --recording {cafe,basements,studies,office_kitchens,playroooms}
|
|
```
|
|
|
|
If you wish to customize it, explore additional features, or save it use the CLI with the `--help` option for guidance:
|
|
|
|
```bash
|
|
python -m rgbd --help
|
|
```
|