wehub/rerun-io--rerun
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 13:05:14 +08:00
commit 2a547be7fe
7904 changed files with 1000926 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
examples/python/objectron/README.md
+134
View File
@@ -0,0 +1,134 @@
<!--[metadata]
title = "Objectron"
tags = ["2D", "3D", "Object detection", "Pinhole camera", "Blueprint"]
thumbnail = "https://static.rerun.io/objectron/b645ef3c8eff33fbeaefa6d37e0f9711be15b202/480w.png"
thumbnail_dimensions = [480, 480]
# Channel = "release" - disabled because it sometimes have bad first-frame heuristics
build_args = ["--frames=150"]
-->
Visualize the [Google Research Objectron](https://github.com/google-research-datasets/Objectron) dataset including camera poses, sparse point-clouds and surfaces characterization.
<picture>
<source media="(max-width: 480px)" srcset="https://static.rerun.io/objectron/8ea3a37e6b4af2e06f8e2ea5e70c1951af67fea8/480w.png">
<source media="(max-width: 768px)" srcset="https://static.rerun.io/objectron/8ea3a37e6b4af2e06f8e2ea5e70c1951af67fea8/768w.png">
<source media="(max-width: 1024px)" srcset="https://static.rerun.io/objectron/8ea3a37e6b4af2e06f8e2ea5e70c1951af67fea8/1024w.png">
<source media="(max-width: 1200px)" srcset="https://static.rerun.io/objectron/8ea3a37e6b4af2e06f8e2ea5e70c1951af67fea8/1200w.png">
<img src="https://static.rerun.io/objectron/8ea3a37e6b4af2e06f8e2ea5e70c1951af67fea8/full.png" alt="Objectron example screenshot">
</picture>
## Used Rerun types
[`Points3D`](https://www.rerun.io/docs/reference/types/archetypes/points3d), [`Boxes3D`](https://www.rerun.io/docs/reference/types/archetypes/boxes3d), [`EncodedImage`](https://www.rerun.io/docs/reference/types/archetypes/encoded_image), [`Transform3D`](https://www.rerun.io/docs/reference/types/archetypes/transform3d), [`Pinhole`](https://www.rerun.io/docs/reference/types/archetypes/pinhole)
## Background
This example visualizes the Objectron database, a rich collection of object-centric video clips accompanied by AR session metadata.
With high-resolution images, object pose, camera pose, point-cloud, and surface plane information available for each sample, the visualization offers a comprehensive view of the object from various angles.
Additionally, the dataset provides manually annotated 3D bounding boxes, enabling precise object localization and orientation.
## Logging and visualizing with Rerun
The visualizations in this example were created with the following Rerun code:
### Timelines
For each processed frame, all data sent to Rerun is associated with the two [`timelines`](https://www.rerun.io/docs/concepts/logging-and-ingestion/timelines) `time` and `frame_idx`.
```python
rr.set_time("frame", sequence=sample.index)
rr.set_time("time", duration=sample.timestamp)
```
### Video
Pinhole camera is utilized for achieving a 3D view and camera perspective through the use of the [`Pinhole`](https://www.rerun.io/docs/reference/types/archetypes/pinhole) and [`Transform3D`](https://www.rerun.io/docs/reference/types/archetypes/transform3d) archetypes.
```python
rr.log(
"world/camera",
rr.Transform3D(translation=translation, rotation=rr.Quaternion(xyzw=rot.as_quat())),
)
```
```python
rr.log(
"world/camera",
rr.Pinhole(
resolution=[w, h],
image_from_camera=intrinsics,
camera_xyz=rr.ViewCoordinates.RDF,
),
)
```
The input video is logged as a sequence of [`EncodedImage`](https://www.rerun.io/docs/reference/types/archetypes/encoded_image) objects to the `world/camera` entity.
```python
rr.log("world/camera", rr.EncodedImage(path=sample.image_path))
```
### Sparse point clouds
Sparse point clouds from `ARFrame` are logged as [`Points3D`](https://www.rerun.io/docs/reference/types/archetypes/points3d) archetype to the `world/points` entity.
```python
rr.log("world/points", rr.Points3D(positions, colors=[255, 255, 255, 255]))
```
### Annotated bounding boxes
Bounding boxes annotated from `ARFrame` are logged as [`Boxes3D`](https://www.rerun.io/docs/reference/types/archetypes/boxes3d), containing details such as object position, sizes, center and rotation.
```python
rr.log(
f"world/annotations/box-{bbox.id}",
rr.Boxes3D(
half_sizes=0.5 * np.array(bbox.scale),
centers=bbox.translation,
rotations=rr.Quaternion(xyzw=rot.as_quat()),
colors=[160, 230, 130, 255],
labels=bbox.category,
),
static=True,
)
```
### Setting up the default blueprint
The default blueprint is configured with the following code:
```python
blueprint = rrb.Horizontal(
rrb.Spatial3DView(origin="/world", name="World"),
rrb.Spatial2DView(origin="/world/camera", name="Camera", contents=["/world/**"]),
)
```
In particular, we want to reproject the points and the 3D annotation box in the 2D camera view corresponding to the pinhole logged at `"/world/camera"`. This is achieved by setting the view's contents to the entire `"/world/**"` subtree, which include both the pinhole transform and the image data, as well as the point cloud and the 3D annotation box.
## Run the code
To run this example, make sure you have the [required Python version](https://ref.rerun.io/docs/python/main/common#supported-python-versions), the Rerun repository checked out and the latest SDK installed:
```bash
pip install --upgrade rerun-sdk # install the latest Rerun SDK
git clone git@github.com:rerun-io/rerun.git # Clone the repository
cd rerun
git checkout latest # Check out the commit matching the latest SDK release
```
Install the necessary libraries specified in the requirements file:
```bash
pip install -e examples/python/objectron
```
To experiment with the provided example, simply execute the main Python script:
```bash
python -m objectron # run the example
```
You can specify the objectron recording:
```bash
python -m objectron --recording {bike,book,bottle,camera,cereal_box,chair,cup,laptop,shoe}
```
If you wish to customize it, explore additional features, or save it use the CLI with the `--help` option for guidance:
```bash
python -m objectron --help
```
examples/python/objectron/objectron/__init__.py
View File
examples/python/objectron/objectron/__main__.py
Executable
+252
View File
@@ -0,0 +1,252 @@
#!/usr/bin/env python3
"""Example of using the Rerun SDK to log the Objectron dataset."""
from __future__ import annotations
import argparse
import logging
import math
import os
import sys
import time
from dataclasses import dataclass
from pathlib import Path
from typing import TYPE_CHECKING
import numpy as np
from scipy.spatial.transform import Rotation as R
import rerun as rr # pip install rerun-sdk
import rerun.blueprint as rrb
from .download_dataset import (
ANNOTATIONS_FILENAME,
AVAILABLE_RECORDINGS,
GEOMETRY_FILENAME,
LOCAL_DATASET_DIR,
ensure_recording_available,
)
from .proto.objectron.proto import ARCamera, ARFrame, ARPointCloud, Object, ObjectType, Sequence
if TYPE_CHECKING:
from collections.abc import Iterable, Iterator
@dataclass
class SampleARFrame:
"""An `ARFrame` sample and the relevant associated metadata."""
index: int
timestamp: float
dirpath: Path
frame: ARFrame
image_path: Path
def read_ar_frames(
dirpath: Path,
num_frames: int,
run_forever: bool,
per_frame_sleep: float,
) -> Iterator[SampleARFrame]:
"""
Loads up to `num_frames` consecutive ARFrames from the given path on disk.
`dirpath` should be of the form `dataset/bike/batch-8/16/`.
"""
path = dirpath / GEOMETRY_FILENAME
print(f"loading ARFrames from {path}")
time_offset = 0
frame_offset = 0
while True:
frame_idx = 0
data = Path(path).read_bytes()
while len(data) > 0 and frame_idx < num_frames:
next_len = int.from_bytes(data[:4], byteorder="little", signed=False)
data = data[4:]
frame = ARFrame().parse(data[:next_len])
img_path = Path(os.path.join(dirpath, f"video/{frame_idx}.jpg"))
yield SampleARFrame(
index=frame_idx + frame_offset,
timestamp=frame.timestamp + time_offset,
dirpath=dirpath,
frame=frame,
image_path=img_path,
)
data = data[next_len:]
frame_idx += 1
if run_forever and per_frame_sleep > 0.0:
time.sleep(per_frame_sleep)
if run_forever:
time_offset += frame.timestamp
frame_offset += frame_idx
else:
break
def read_annotations(dirpath: Path) -> Sequence:
"""
Loads the annotations from the given path on disk.
`dirpath` should be of the form `dataset/bike/batch-8/16/`.
"""
path = dirpath / ANNOTATIONS_FILENAME
print(f"loading annotations from {path}")
data = Path(path).read_bytes()
seq = Sequence().parse(data)
return seq
def log_ar_frames(samples: Iterable[SampleARFrame], seq: Sequence) -> None:
"""Logs a stream of `ARFrame` samples and their annotations with the Rerun SDK."""
rr.log("world", rr.ViewCoordinates.RIGHT_HAND_Y_UP, static=True)
log_annotated_bboxes(seq.objects)
frame_times = []
for sample in samples:
rr.set_time("frame", sequence=sample.index)
rr.set_time("time", duration=sample.timestamp)
frame_times.append(sample.timestamp)
rr.log("world/camera", rr.EncodedImage(path=sample.image_path))
log_camera(sample.frame.camera)
log_point_cloud(sample.frame.raw_feature_points)
def log_camera(cam: ARCamera) -> None:
"""Logs a camera from an `ARFrame` using the Rerun SDK."""
X = np.asarray([1.0, 0.0, 0.0])
Z = np.asarray([0.0, 0.0, 1.0])
world_from_cam = np.asarray(cam.transform).reshape((4, 4))
translation = world_from_cam[0:3, 3]
intrinsics = np.asarray(cam.intrinsics).reshape((3, 3))
rot = R.from_matrix(world_from_cam[0:3, 0:3])
(w, h) = (cam.image_resolution_width, cam.image_resolution_height)
# Because the dataset was collected in portrait:
swizzle_x_y = np.asarray([[0, 1, 0], [1, 0, 0], [0, 0, 1]])
intrinsics = swizzle_x_y @ intrinsics @ swizzle_x_y
rot = rot * R.from_rotvec((math.tau / 4.0) * Z)
(w, h) = (h, w)
rot = rot * R.from_rotvec((math.tau / 2.0) * X) # TODO(emilk): figure out why this is needed
rr.log(
"world/camera",
rr.Transform3D(translation=translation, rotation=rr.Quaternion(xyzw=rot.as_quat())),
)
rr.log(
"world/camera",
rr.Pinhole(
resolution=[w, h],
image_from_camera=intrinsics,
camera_xyz=rr.ViewCoordinates.RDF,
),
)
def log_point_cloud(point_cloud: ARPointCloud) -> None:
"""Logs a point cloud from an `ARFrame` using the Rerun SDK."""
positions = np.array([[p.x, p.y, p.z] for p in point_cloud.point]).astype(np.float32)
rr.log("world/points", rr.Points3D(positions, colors=[255, 255, 255, 255]))
def log_annotated_bboxes(bboxes: Iterable[Object]) -> None:
"""Logs all the bounding boxes annotated in an `ARFrame` sequence using the Rerun SDK."""
for bbox in bboxes:
if bbox.type != ObjectType.BOUNDING_BOX:
print(f"err: object type not supported: {bbox.type}")
continue
rot = R.from_matrix(np.asarray(bbox.rotation).reshape((3, 3)))
rr.log(
f"world/annotations/box-{bbox.id}",
rr.Boxes3D(
half_sizes=0.5 * np.array(bbox.scale),
centers=bbox.translation,
rotations=rr.Quaternion(xyzw=rot.as_quat()),
colors=[160, 230, 130, 255],
labels=bbox.category,
),
static=True,
)
def main() -> None:
# Ensure the logging in download_dataset.py gets written to stderr:
logging.getLogger().addHandler(logging.StreamHandler())
logging.getLogger().setLevel("INFO")
parser = argparse.ArgumentParser(description="Logs Objectron data using the Rerun SDK.")
parser.add_argument(
"--frames",
type=int,
default=sys.maxsize,
help="If specified, limits the number of frames logged",
)
parser.add_argument("--run-forever", action="store_true", help="Run forever, continually logging data.")
parser.add_argument(
"--per-frame-sleep",
type=float,
default=0.1,
help="Sleep this much for each frame read, if --run-forever",
)
parser.add_argument(
"--recording",
type=str,
choices=AVAILABLE_RECORDINGS,
default=AVAILABLE_RECORDINGS[1],
help="The objectron recording to log to Rerun.",
)
parser.add_argument(
"--force-reprocess-video",
action="store_true",
help="Reprocess video frames even if they already exist",
)
parser.add_argument(
"--dataset-dir",
type=Path,
default=LOCAL_DATASET_DIR,
help="Directory to save example videos to.",
)
rr.script_add_args(parser)
args = parser.parse_args()
blueprint = rrb.Horizontal(
rrb.Spatial3DView(origin="/world", name="World"),
rrb.Spatial2DView(origin="/world/camera", name="Camera", contents=["/world/**"]),
)
rr.script_setup(
args,
"rerun_example_objectron",
default_blueprint=blueprint,
)
dir = ensure_recording_available(args.recording, args.dataset_dir, args.force_reprocess_video)
samples = read_ar_frames(dir, args.frames, args.run_forever, args.per_frame_sleep)
seq = read_annotations(dir)
log_ar_frames(samples, seq)
rr.script_teardown(args)
if __name__ == "__main__":
main()
examples/python/objectron/objectron/download_dataset.py
+109
View File
@@ -0,0 +1,109 @@
from __future__ import annotations
import logging
import os
from pathlib import Path
from typing import Final
import cv2
import requests
DATASET_BASE_URL = "https://storage.googleapis.com/objectron"
LOCAL_DATASET_DIR: Final = Path(__file__).parent.parent / "dataset"
IMAGE_RESOLUTION: Final = (1440, 1920)
GEOMETRY_FILENAME: Final = "geometry.pbdata"
ANNOTATIONS_FILENAME: Final = "annotation.pbdata"
VIDEO_FILENAME: Final = "video.MOV"
AVAILABLE_RECORDINGS = [
"bike",
"book",
"bottle",
"camera",
"cereal_box",
"chair",
"cup",
"laptop",
"shoe",
]
def ensure_downloaded(src_url: str, dst_path: Path) -> None:
os.makedirs(dst_path.parent, exist_ok=True)
if not dst_path.exists():
logging.info("Downloading %s to %s", src_url, dst_path)
with requests.get(src_url, stream=True) as req:
req.raise_for_status()
with open(dst_path, "wb") as f:
f.writelines(req.iter_content(chunk_size=8192))
def find_path_if_downloaded(recording_name: str, local_dataset_dir: Path) -> Path | None:
local_recording_dir = local_dataset_dir / recording_name
paths = list(local_recording_dir.glob(f"**/{ANNOTATIONS_FILENAME}"))
if paths:
return paths[0].parent
return None
def get_recording_id_from_name(recording_name: str) -> str:
recording_ids_raw = requests.get(f"{DATASET_BASE_URL}/v1/index/{recording_name}_annotations_test").text
recording_id = recording_ids_raw.split("\n")[0]
return recording_id
def ensure_opencv_version_ok() -> None:
if cv2.getVersionMajor() == 4 and cv2.getVersionMinor() == 6:
raise RuntimeError(
"""Opencv 4.6 contains a bug which will unpack some videos with the incorrect orientation.
See: https://github.com/opencv/opencv/issues/22088
Please upgrade or downgrade as appropriate.""",
)
def ensure_recording_downloaded(recording_name: str, dataset_dir: Path) -> Path:
"""
Makes sure the recording is downloaded.
Returns the path to where the dataset is downloaded locally.
"""
ensure_opencv_version_ok()
local_recording_dir = find_path_if_downloaded(recording_name, dataset_dir)
if local_recording_dir is not None:
return local_recording_dir
recording_id = get_recording_id_from_name(recording_name)
local_recording_dir = dataset_dir / recording_id
recording_url = f"{DATASET_BASE_URL}/videos/{recording_id}"
ensure_downloaded(f"{recording_url}/{VIDEO_FILENAME}", local_recording_dir / VIDEO_FILENAME)
ensure_downloaded(f"{recording_url}/{GEOMETRY_FILENAME}", local_recording_dir / GEOMETRY_FILENAME)
ensure_downloaded(
f"{DATASET_BASE_URL}/annotations/{recording_id}.pbdata",
local_recording_dir / ANNOTATIONS_FILENAME,
)
return local_recording_dir
def ensure_video_is_split_into_frames(recording_dir: Path, force_reprocess: bool = False) -> None:
video_path = recording_dir / VIDEO_FILENAME
frames_dir = recording_dir / "video"
if force_reprocess or not frames_dir.exists():
logging.info("Splitting video at %s into frames in %s", video_path, frames_dir)
os.makedirs(frames_dir, exist_ok=True)
vidcap = cv2.VideoCapture(str(video_path))
success, image = vidcap.read()
count = 0
while success:
cv2.imwrite(f"{frames_dir}/{count}.jpg", image)
success, image = vidcap.read()
count += 1
def ensure_recording_available(name: str, local_dataset_dir: Path, force_reprocess_video: bool = False) -> Path:
recording_path = ensure_recording_downloaded(name, local_dataset_dir)
ensure_video_is_split_into_frames(recording_path, force_reprocess_video)
return recording_path
examples/python/objectron/objectron/proto/LICENSE.txt
+49
View File
@@ -0,0 +1,49 @@
All .proto files come from https://github.com/google-research-datasets/Objectron under the following license:
# Computational Use of Data Agreement v1.0
This is the Computational Use of Data Agreement, Version 1.0 (the “C-UDA”). Capitalized terms are defined in Section 5. Data Provider and you agree as follows:
1. **Provision of the Data**
1.1. You may use, modify, and distribute the Data made available to you by the Data Provider under this C-UDA for Computational Use if you follow the C-UDA's terms.
1.2. Data Provider will not sue you or any Downstream Recipient for any claim arising out of the use, modification, or distribution of the Data provided you meet the terms of the C-UDA.
1.3 This C-UDA does not restrict your use, modification, or distribution of any portions of the Data that are in the public domain or that may be used, modified, or distributed under any other legal exception or limitation.
2. **Restrictions**
2.1 You agree that you will use the Data solely for Computational Use.
2.2 The C-UDA does not impose any restriction with respect to the use, modification, or distribution of Results.
3. **Redistribution of Data**
3.1. You may redistribute the Data, so long as:
3.1.1. You include with any Data you redistribute all credit or attribution information that you received with the Data, and your terms require any Downstream Recipient to do the same; and
3.1.2. You bind each recipient to whom you redistribute the Data to the terms of the C-UDA.
4. **No Warranty, Limitation of Liability**
4.1. Data Provider does not represent or warrant that it has any rights whatsoever in the Data.
4.2. THE DATA IS PROVIDED ON AN “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED INCLUDING, WITHOUT LIMITATION, ANY WARRANTIES OR CONDITIONS OF TITLE, NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
4.3. NEITHER DATA PROVIDER NOR ANY UPSTREAM DATA PROVIDER SHALL HAVE ANY LIABILITY FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING WITHOUT LIMITATION LOST PROFITS), HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE DATA OR RESULTS, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
5. **Definitions**
5.1. “Computational Use” means activities necessary to enable the use of Data (alone or along with other material) for analysis by a computer.
5.2.“Data” means the material you receive under the C-UDA in modified or unmodified form, but not including Results.
5.3. “Data Provider” means the source from which you receive the Data and with whom you enter into the C-UDA.
5.4. “Downstream Recipient” means any person or persons who receives the Data directly or indirectly from you in accordance with the C-UDA.
5.5. “Result” means anything that you develop or improve from your use of Data that does not include more than a de minimis portion of the Data on which the use is based. Results may include de minimis portions of the Data necessary to report on or explain use that has been conducted with the Data, such as figures in scientific papers, but do not include more. Artificial intelligence models trained on Data (and which do not include more than a de minimis portion of Data) are Results.
examples/python/objectron/objectron/proto/__init__.py
View File
examples/python/objectron/objectron/proto/a_r_capture_metadata.proto
+621
View File
@@ -0,0 +1,621 @@
syntax = "proto2";
package objectron.proto;
option objc_class_prefix = "XNOPB";
// Info about the camera characteristics used to capture images and depth data.
message AVCameraCalibrationData {
// 3x3 row-major matrix relating a camera's internal properties to an ideal
// pinhole-camera model.
repeated float intrinsic_matrix = 1 [packed = true];
// The image dimensions to which the intrinsic_matrix values are relative.
optional float intrinsic_matrix_reference_dimension_width = 2;
optional float intrinsic_matrix_reference_dimension_height = 3;
// 3x4 row-major matrix relating a camera's position and orientation to a
// world or scene coordinate system. Consists of a unitless 3x3 rotation
// matrix (R) on the left and a translation (t) 3x1 vector on the right. The
// translation vector's units are millimeters. For example:
//
// |r1,1 r2,1 r3,1 | t1|
// [R | t] = |r1,2 r2,2 r3,2 | t2|
// |r1,3 r2,3 r3,3 | t3|
//
// is stored as [r11, r21, r31, t1, r12, r22, r32, t2, ...]
repeated float extrinsic_matrix = 4 [packed = true];
// The size, in millimeters, of one image pixel.
optional float pixel_size = 5;
// A list of floating-point values describing radial distortions imparted by
// the camera lens, for use in rectifying camera images.
repeated float lens_distortion_lookup_values = 6 [packed = true];
// A list of floating-point values describing radial distortions for use in
// reapplying camera geometry to a rectified image.
repeated float inverse_lens_distortion_lookup_values = 7 [packed = true];
// The offset of the distortion center of the camera lens from the top-left
// corner of the image.
optional float lens_distortion_center_x = 8;
optional float lens_distortion_center_y = 9;
}
// Container for depth data information.
message AVDepthData {
// PNG representation of the grayscale depth data map. See discussion about
// depth_data_map_original_minimum_value, below, for information about how
// to interpret the pixel values.
optional bytes depth_data_map = 1;
// Pixel format type of the original captured depth data.
optional string depth_data_type = 2;
// Indicates the general accuracy of the depth_data_map.
enum Accuracy {
UNDEFINED_ACCURACY = 0;
// Values in the depth map are usable for foreground/background separation
// but are not absolutely accurate in the physical world.
RELATIVE = 1;
// Values in the depth map are absolutely accurate in the physical world.
ABSOLUTE = 2;
}
optional Accuracy depth_data_accuracy = 3 [default = RELATIVE];
// Indicates whether the depth_data_map contains temporally smoothed data.
optional bool depth_data_filtered = 4;
// Quality of the depth_data_map.
enum Quality {
UNDEFINED_QUALITY = 0;
HIGH = 1;
LOW = 2;
}
optional Quality depth_data_quality = 5;
// Associated calibration data for the depth_data_map.
optional AVCameraCalibrationData camera_calibration_data = 6;
// The original range of values expressed by the depth_data_map, before
// grayscale normalization. For example, if the minimum and maximum values
// indicate a range of [0.5, 2.2], and the depth_data_type value indicates
// it was a depth map, then white pixels (255, 255, 255) will map to 0.5 and
// black pixels (0, 0, 0) will map to 2.2 with the grayscale range linearly
// interpolated inbetween. Conversely, if the depth_data_type value indicates
// it was a disparity map, then white pixels will map to 2.2 and black pixels
// will map to 0.5.
optional float depth_data_map_original_minimum_value = 7;
optional float depth_data_map_original_maximum_value = 8;
// The width of the depth buffer map.
optional int32 depth_data_map_width = 9;
// The height of the depth buffer map.
optional int32 depth_data_map_height = 10;
// The row-major flattened array of the depth buffer map pixels. This will be
// either a float32 or float16 byte array, depending on 'depth_data_type'.
optional bytes depth_data_map_raw_values = 11;
}
// Estimated scene lighting information associated with a captured video frame.
message ARLightEstimate {
// The estimated intensity, in lumens, of ambient light throughout the scene.
optional double ambient_intensity = 1;
// The estimated color temperature, in degrees Kelvin, of ambient light
// throughout the scene.
optional double ambient_color_temperature = 2;
// Data describing the estimated lighting environment in all directions.
// Second-level spherical harmonics in separate red, green, and blue data
// planes. Thus, this buffer contains 3 sets of 9 coefficients, or a total of
// 27 values.
repeated float spherical_harmonics_coefficients = 3 [packed = true];
message DirectionVector {
optional float x = 1;
optional float y = 2;
optional float z = 3;
}
// A vector indicating the orientation of the strongest directional light
// source, normalized in the world-coordinate space.
optional DirectionVector primary_light_direction = 4;
// The estimated intensity, in lumens, of the strongest directional light
// source in the scene.
optional float primary_light_intensity = 5;
}
// Information about the camera position and imaging characteristics for a
// captured video frame.
message ARCamera {
// The general quality of position tracking available when the camera captured
// a frame.
enum TrackingState {
UNDEFINED_TRACKING_STATE = 0;
// Camera position tracking is not available.
UNAVAILABLE = 1;
// Tracking is available, but the quality of results is questionable.
LIMITED = 2;
// Camera position tracking is providing optimal results.
NORMAL = 3;
}
optional TrackingState tracking_state = 1 [default = UNAVAILABLE];
// A possible diagnosis for limited position tracking quality as of when the
// frame was captured.
enum TrackingStateReason {
UNDEFINED_TRACKING_STATE_REASON = 0;
// The current tracking state is not limited.
NONE = 1;
// Not yet enough camera or motion data to provide tracking information.
INITIALIZING = 2;
// The device is moving too fast for accurate image-based position tracking.
EXCESSIVE_MOTION = 3;
// Not enough distinguishable features for image-based position tracking.
INSUFFICIENT_FEATURES = 4;
// Tracking is limited due to a relocalization in progress.
RELOCALIZING = 5;
}
optional TrackingStateReason tracking_state_reason = 2 [default = NONE];
// 4x4 row-major matrix expressing position and orientation of the camera in
// world coordinate space.
repeated float transform = 3 [packed = true];
// The orientation of the camera, expressed as roll, pitch, and yaw values.
message EulerAngles {
optional float roll = 1;
optional float pitch = 2;
optional float yaw = 3;
}
optional EulerAngles euler_angles = 4;
// The width and height, in pixels, of the captured camera image.
optional int32 image_resolution_width = 5;
optional int32 image_resolution_height = 6;
// 3x3 row-major matrix that converts between the 2D camera plane and 3D world
// coordinate space.
repeated float intrinsics = 7 [packed = true];
// 4x4 row-major transform matrix appropriate for rendering 3D content to
// match the image captured by the camera.
repeated float projection_matrix = 8 [packed = true];
// 4x4 row-major transform matrix appropriate for converting from world-space
// to camera space. Relativized for the captured_image orientation (i.e.
// UILandscapeOrientationRight).
repeated float view_matrix = 9 [packed = true];
}
// Container for a 3D mesh describing face topology.
message ARFaceGeometry {
// Each vertex represents a 3D point in the face mesh, in the face coordinate
// space.
message Vertex {
optional float x = 1;
optional float y = 2;
optional float z = 3;
}
repeated Vertex vertices = 1;
// The number of elements in the vertices list.
optional int32 vertex_count = 2;
// Each texture coordinate represents UV texture coordinates for the vertex at
// the corresponding index in the vertices buffer.
message TextureCoordinate {
optional float u = 1;
optional float v = 2;
}
repeated TextureCoordinate texture_coordinates = 3;
// The number of elements in the texture_coordinates list.
optional int32 texture_coordinate_count = 4;
// Each integer value in this ordered list represents an index into the
// vertices and texture_coordinates lists. Each set of three indices
// identifies the vertices comprising a single triangle in the mesh. Each set
// of three indices forms a triangle, so the number of indices in the
// triangle_indices buffer is three times the triangle_count value.
repeated int32 triangle_indices = 5 [packed = true];
// The number of triangles described by the triangle_indices buffer.
optional int32 triangle_count = 6;
}
// Contains a list of blend shape entries wherein each item maps a specific
// blend shape location to its associated coefficient.
message ARBlendShapeMap {
message MapEntry {
// Identifier for the specific facial feature.
optional string blend_shape_location = 1;
// Indicates the current position of the feature relative to its neutral
// configuration, ranging from 0.0 (neutral) to 1.0 (maximum movement).
optional float blend_shape_coefficient = 2;
}
repeated MapEntry entries = 1;
}
// Information about the pose, topology, and expression of a detected face.
message ARFaceAnchor {
// A coarse triangle mesh representing the topology of the detected face.
optional ARFaceGeometry geometry = 1;
// A map of named coefficients representing the detected facial expression in
// terms of the movement of specific facial features.
optional ARBlendShapeMap blend_shapes = 2;
// 4x4 row-major matrix encoding the position, orientation, and scale of the
// anchor relative to the world coordinate space.
repeated float transform = 3;
// Indicates whether the anchor's transform is valid. Frames that have a face
// anchor with this value set to NO should probably be ignored.
optional bool is_tracked = 4;
}
// Container for a 3D mesh.
message ARPlaneGeometry {
message Vertex {
optional float x = 1;
optional float y = 2;
optional float z = 3;
}
// Each texture coordinate represents UV texture coordinates for the vertex at
// the corresponding index in the vertices buffer.
message TextureCoordinate {
optional float u = 1;
optional float v = 2;
}
// A buffer of vertex positions for each point in the plane mesh.
repeated Vertex vertices = 1;
// The number of elements in the vertices buffer.
optional int32 vertex_count = 2;
// A buffer of texture coordinate values for each point in the plane mesh.
repeated TextureCoordinate texture_coordinates = 3;
// The number of elements in the texture_coordinates buffer.
optional int32 texture_coordinate_count = 4;
// Each integer value in this ordered list represents an index into the
// vertices and texture_coordinates lists. Each set of three indices
// identifies the vertices comprising a single triangle in the mesh. Each set
// of three indices forms a triangle, so the number of indices in the
// triangle_indices buffer is three times the triangle_count value.
repeated int32 triangle_indices = 5 [packed = true];
// Each set of three indices forms a triangle, so the number of indices in the
// triangle_indices buffer is three times the triangle_count value.
optional int32 triangle_count = 6;
// Each value in this buffer represents the position of a vertex along the
// boundary polygon of the estimated plane. The owning plane anchor's
// transform matrix defines the coordinate system for these points.
repeated Vertex boundary_vertices = 7;
// The number of elements in the boundary_vertices buffer.
optional int32 boundary_vertex_count = 8;
}
// Information about the position and orientation of a real-world flat surface.
message ARPlaneAnchor {
enum Alignment {
UNDEFINED = 0;
// The plane is perpendicular to gravity.
HORIZONTAL = 1;
// The plane is parallel to gravity.
VERTICAL = 2;
}
// Wrapper for a 3D point / vector within the plane. See extent and center
// values for more information.
message PlaneVector {
optional float x = 1;
optional float y = 2;
optional float z = 3;
}
enum PlaneClassification {
NONE = 0;
WALL = 1;
FLOOR = 2;
CEILING = 3;
TABLE = 4;
SEAT = 5;
}
// The classification status for the plane.
enum PlaneClassificationStatus {
// The classification process for the plane anchor has completed but the
// result is inconclusive.
UNKNOWN = 0;
// No classification information can be provided (set on error or if the
// device does not support plane classification).
UNAVAILABLE = 1;
// The classification process has not completed.
UNDETERMINED = 2;
// The classification process for the plane anchor has completed.
KNOWN = 3;
}
// The ID of the plane.
optional string identifier = 1;
// 4x4 row-major matrix encoding the position, orientation, and scale of the
// anchor relative to the world coordinate space.
repeated float transform = 2;
// The general orientation of the detected plane with respect to gravity.
optional Alignment alignment = 3;
// A coarse triangle mesh representing the general shape of the detected
// plane.
optional ARPlaneGeometry geometry = 4;
// The center point of the plane relative to its anchor position.
// Although the type of this property is a 3D vector, a plane anchor is always
// two-dimensional, and is always positioned in only the x and z directions
// relative to its transform position. (That is, the y-component of this
// vector is always zero.)
optional PlaneVector center = 5;
// The estimated width and length of the detected plane.
optional PlaneVector extent = 6;
// A Boolean value that indicates whether plane classification is available on
// the current device. On devices without plane classification support, all
// plane anchors report a classification value of NONE
// and a classification_status value of UNAVAILABLE.
optional bool classification_supported = 7;
// A general characterization of what kind of real-world surface the plane
// anchor represents.
optional PlaneClassification classification = 8;
// The current state of process for classifying the plane anchor.
// When this property's value is KNOWN, the classification property represents
// characterization of the real-world surface corresponding to the
// plane anchor.
optional PlaneClassificationStatus classification_status = 9;
}
// A collection of points in the world coordinate space.
message ARPointCloud {
message Point {
optional float x = 1;
optional float y = 2;
optional float z = 3;
}
// The number of points in the cloud.
optional int32 count = 1;
// The list of detected points.
repeated Point point = 2;
// A list of unique identifiers corresponding to detected feature points.
// Each identifier in this list corresponds to the point at the same index
// in the points array.
repeated int64 identifier = 3 [packed = true];
}
// A 3D vector
message CMVector {
optional double x = 1;
optional double y = 2;
optional double z = 3;
}
// Represents calibrated magnetic field data and accuracy estimate of it.
message CMCalibratedMagneticField {
// Indicates the calibration accuracy of a magnetic field estimate.
enum CalibrationAccuracy {
UNCALIBRATED = 0;
LOW = 1;
MEDIUM = 2;
HIGH = 3;
}
// Vector of magnetic field estimate.
optional CMVector field = 1;
// Calibration accuracy of a magnetic field estimate.
optional CalibrationAccuracy calibration_accuracy = 2;
}
// A sample of device motion data.
// Encapsulates measurements of the attitude, rotation rate, magnetic field, and
// acceleration of the device. Core Media applies different algorithms of
// bias-reduction and stabilization to rotation rate, magnetic field and
// acceleration values. For raw values check correspondent fields in
// CMMotionManagerSnapshot object.
message CMDeviceMotion {
message Quaternion {
optional double x = 1;
optional double y = 2;
optional double z = 3;
optional double w = 4;
}
// The device motion data object timestamp. May differ from the frame
// timestamp value since the data may be collected at higher rate.
optional double timestamp = 1;
// The quaternion representing the device’s orientation relative to a known
// frame of reference at a point in time.
optional Quaternion attitude_quaternion = 2;
// The gravity acceleration vector expressed in the device's reference frame.
optional CMVector gravity = 3;
// The acceleration that the user is giving to the device.
optional CMVector user_acceleration = 4;
// Returns the magnetic field vector filtered with respect to the device bias.
optional CMCalibratedMagneticField magnetic_field = 5;
// The rotation rate of the device adjusted by bias-removing Core Motion
// algorithms.
optional CMVector rotation_rate = 6;
}
// A sample of raw accelerometer data.
message CMAccelerometerData {
// The accelerometer data object timestamp. May differ from the frame
// timestamp value since the data may be collected at higher rate.
optional double timestamp = 1;
// Raw acceleration measured by the accelerometer which effectively is a sum
// of gravity and user_acceleration of CMDeviceMotion object.
optional CMVector acceleration = 2;
}
// A sample of raw gyroscope data.
message CMGyroData {
// The gyroscope data object timestamp. May differ from the frame
// timestamp value since the data may be collected at higher rate.
optional double timestamp = 1;
// Raw rotation rate as measured by the gyroscope.
optional CMVector rotation_rate = 2;
}
// A sample of raw magnetometer data.
message CMMagnetometerData {
// The magnetometer data object timestamp. May differ from the frame
// timestamp value since the data may be collected at higher rate.
optional double timestamp = 1;
// Raw magnetic field measured by the magnetometer.
optional CMVector magnetic_field = 2;
}
// Contains most recent snapshots of device motion data
message CMMotionManagerSnapshot {
// Most recent samples of device motion data.
repeated CMDeviceMotion device_motion = 1;
// Most recent samples of raw accelerometer data.
repeated CMAccelerometerData accelerometer_data = 2;
// Most recent samples of raw gyroscope data.
repeated CMGyroData gyro_data = 3;
// Most recent samples of raw magnetometer data.
repeated CMMagnetometerData magnetometer_data = 4;
}
// Video image and face position tracking information.
message ARFrame {
// The timestamp for the frame.
optional double timestamp = 1;
// The depth data associated with the frame. Not all frames have depth data.
optional AVDepthData depth_data = 2;
// The depth data object timestamp associated with the frame. May differ from
// the frame timestamp value. Is only set when the frame has depth_data.
optional double depth_data_timestamp = 3;
// Camera information associated with the frame.
optional ARCamera camera = 4;
// Light information associated with the frame.
optional ARLightEstimate light_estimate = 5;
// Face anchor information associated with the frame. Not all frames have an
// active face anchor.
optional ARFaceAnchor face_anchor = 6;
// Plane anchors associated with the frame. Not all frames have a plane
// anchor. Plane anchors and face anchors are mutually exclusive.
repeated ARPlaneAnchor plane_anchor = 7;
// The current intermediate results of the scene analysis used to perform
// world tracking.
optional ARPointCloud raw_feature_points = 8;
// Snapshot of Core Motion CMMotionManager object containing most recent
// motion data associated with the frame. Since motion data capture rates can
// be higher than rates of AR capture, the entities of this object reflect all
// of the aggregated events which have occurred since the last ARFrame was
// recorded.
optional CMMotionManagerSnapshot motion_manager_snapshot = 9;
}
// Mesh geometry data stored in an array-based format.
message ARMeshGeometry {
message Vertex {
optional float x = 1;
optional float y = 2;
optional float z = 3;
}
message Face {
// Indices of vertices defining the face from correspondent array of parent
// message. A typical face is triangular.
repeated int32 vertex_indices = 1 [packed = true];
}
// Type of objects
enum MeshClassification {
NONE = 0;
WALL = 1;
FLOOR = 2;
CEILING = 3;
TABLE = 4;
SEAT = 5;
WINDOW = 6;
DOOR = 7;
}
// The vertices of the mesh.
repeated Vertex vertices = 1;
// The faces of the mesh.
repeated Face faces = 2;
// Rays that define which direction is outside for each face.
// Normals contain 'rays that define which direction is outside for each
// face', in practice the normals count is always identical to vertices count
// which looks like vertices normals and not faces normals.
repeated Vertex normals = 3;
// Classification for each face in the mesh.
repeated MeshClassification classification = 4;
}
// A subdividision of the reconstructed, real-world scene surrounding the user.
message ARMeshAnchor {
// The ID of the mesh.
optional string identifier = 1;
// 4x4 row-major matrix encoding the position, orientation, and scale of the
// anchor relative to the world coordinate space.
repeated float transform = 2 [packed = true];
// 3D information about the mesh such as its shape and classifications.
optional ARMeshGeometry geometry = 3;
}
// Container object for mesh data of real-world scene surrounding the user.
// Even though each ARFrame may have a set of ARMeshAnchors associated with it,
// only a single frame's worth of mesh data is written separately at the end of
// each recording due to concerns regarding latency and memory bloat.
message ARMeshData {
// The timestamp for the data.
optional double timestamp = 1;
// Set of mesh anchors containing the mesh data.
repeated ARMeshAnchor mesh_anchor = 2;
}
examples/python/objectron/objectron/proto/annotation_data.proto
+81
View File
@@ -0,0 +1,81 @@
syntax = "proto3";
package objectron.proto;
import "a_r_capture_metadata.proto";
import "object.proto";
// option cc_api_version = 2;
// option java_api_version = 2;
// Projection of a 3D point on an image, and its metric depth.
message NormalizedPoint2D {
// x-y position of the 2d keypoint in the image coordinate system.
// u,v \in [0, 1], where top left corner is (0, 0) and the bottom-right corner
// is (1, 1).
float x = 1;
float y = 2;
// The depth of the point in the camera coordinate system (in meters).
float depth = 3;
}
// The 3D point in the camera coordinate system, the scales are in meters.
message Point3D {
float x = 1;
float y = 2;
float z = 3;
}
message AnnotatedKeyPoint {
int32 id = 1;
Point3D point_3d = 2;
NormalizedPoint2D point_2d = 3;
}
message ObjectAnnotation {
// Reference to the object identifier in ObjectInstance.
int32 object_id = 1;
// For each objects, list all the annotated keypoints here.
// E.g. for bounding-boxes, we have 8 keypoints, hands = 21 keypoints, etc.
// These normalized points are the projection of the Object's 3D keypoint
// on the current frame's camera poses.
repeated AnnotatedKeyPoint keypoints = 2;
// Visibiity of this annotation in a frame.
float visibility = 3;
}
message FrameAnnotation {
// Unique frame id, corresponds to images.
int32 frame_id = 1;
// List of the annotated objects in this frame. Depending on how many object
// are observable in this frame, we might have non or as much as
// sequence.objects_size() annotations.
repeated ObjectAnnotation annotations = 2;
// Information about the camera transformation (in the world coordinate) and
// imaging characteristics for a captured video frame.
ARCamera camera = 3;
// The timestamp for the frame.
double timestamp = 4;
// Plane center and normal in camera frame.
repeated float plane_center = 5;
repeated float plane_normal = 6;
}
// The sequence protocol contains the annotation data for the entire video clip.
message Sequence {
// List of all the annotated 3D objects in this sequence in the world
// Coordinate system. Given the camera poses of each frame (also in the
// world-coordinate) these objects bounding boxes can be projected to each
// frame to get the per-frame annotation (i.e. image_annotation below).
repeated Object objects = 1;
// List of annotated data per each frame in sequence + frame information.
repeated FrameAnnotation frame_annotations = 2;
}
examples/python/objectron/objectron/proto/object.proto
+114
View File
@@ -0,0 +1,114 @@
syntax = "proto3";
package objectron.proto;
// option cc_api_version = 2;
// option java_api_version = 2;
message KeyPoint {
// The position of the keypoint in the local coordinate system of the rigid
// object.
float x = 1;
float y = 2;
float z = 3;
// Sphere around the keypoint, indicating annotator's confidence of the
// position in meters.
float confidence_radius = 4;
// The name of the keypoint (e.g. legs, head, etc.).
// Does not have to be unique.
string name = 5;
// Indicates whether the keypoint is hidden or not.
bool hidden = 6;
}
message Object {
// Unique object id through a sequence. There might be multiple objects of
// the same label in this sequence.
int32 id = 1;
// Describes what category an object is. E.g. object class, attribute,
// instance or person identity. This provides additional context for the
// object type.
string category = 2;
enum Type {
UNDEFINED_TYPE = 0;
BOUNDING_BOX = 1;
SKELETON = 2;
MESH = 3;
}
Type type = 3;
// 3x3 row-major rotation matrix describing the orientation of the rigid
// object's frame of reference in the world-coordinate system.
repeated float rotation = 4;
// 3x1 vector describing the translation of the rigid object's frame of
// reference in the world-coordinate system in meters.
repeated float translation = 5;
// 3x1 vector describing the scale of the rigid object's frame of reference in
// the world-coordinate system in meters.
repeated float scale = 6;
// List of all the key points associated with this object in the object
// coordinate system.
// The first keypoint is always the object's frame of reference,
// e.g. the centroid of the box.
// E.g. bounding box with its center as frame of reference, the 9 keypoints :
// {0., 0., 0.},
// {-.5, -.5, -.5}, {-.5, -.5, +.5}, {-.5, +.5, -.5}, {-.5, +.5, +.5},
// {+.5, -.5, -.5}, {+.5, -.5, +.5}, {+.5, +.5, -.5}, {+.5, +.5, +.5}
// To get the bounding box in the world-coordinate system, we first scale the
// box then transform the scaled box.
// For example, bounding box in the world coordinate system is
// rotation * scale * keypoints + translation
repeated KeyPoint keypoints = 7;
// Enum to reflect how this object is created.
enum Method {
UNKNOWN_METHOD = 0;
ANNOTATION = 1; // Created by data annotation.
AUGMENTATION = 2; // Created by data augmentation.
}
Method method = 8;
}
// The edge connecting two keypoints together
message Edge {
// keypoint id of the edge's source
int32 source = 1;
// keypoint id of the edge's sink
int32 sink = 2;
}
// The skeleton template for different objects (e.g. humans, chairs, hands, etc)
// The annotation tool reads the skeleton template dictionary.
message Skeleton {
// The origin keypoint in the object coordinate system. (i.e. Point 0, 0, 0)
int32 reference_keypoint = 1;
// The skeleton's category (e.g. human, chair, hand.). Should be unique in the
// dictionary.
string category = 2;
// Initialization value for all the keypoints in the skeleton in the object's
// local coordinate system. Pursuit will transform these points using object's
// transformation to get the keypoint in the world-coordinate.
repeated KeyPoint keypoints = 3;
// List of edges connecting keypoints
repeated Edge edges = 4;
}
// The list of all the modeled skeletons in our library. These models can be
// objects (chairs, desks, etc), humans (full pose, hands, faces, etc), or box.
// We can have multiple skeletons in the same file.
message Skeletons {
repeated Skeleton object = 1;
}
examples/python/objectron/objectron/proto/objectron/__init__.py
View File
examples/python/objectron/objectron/proto/objectron/proto.py
+795
View File
@@ -0,0 +1,795 @@
# Generated by the protocol buffer compiler. DO NOT EDIT!
# sources: a_r_capture_metadata.proto, object.proto, annotation_data.proto
# plugin: python-betterproto
from __future__ import annotations
from dataclasses import dataclass
from typing import List
import betterproto
class AVDepthDataAccuracy(betterproto.Enum):
UNDEFINED_ACCURACY = 0
RELATIVE = 1
ABSOLUTE = 2
class AVDepthDataQuality(betterproto.Enum):
UNDEFINED_QUALITY = 0
HIGH = 1
LOW = 2
class ARCameraTrackingState(betterproto.Enum):
UNDEFINED_TRACKING_STATE = 0
UNAVAILABLE = 1
LIMITED = 2
NORMAL = 3
class ARCameraTrackingStateReason(betterproto.Enum):
UNDEFINED_TRACKING_STATE_REASON = 0
NONE = 1
INITIALIZING = 2
EXCESSIVE_MOTION = 3
INSUFFICIENT_FEATURES = 4
RELOCALIZING = 5
class ARPlaneAnchorAlignment(betterproto.Enum):
UNDEFINED = 0
HORIZONTAL = 1
VERTICAL = 2
class ARPlaneAnchorPlaneClassification(betterproto.Enum):
NONE = 0
WALL = 1
FLOOR = 2
CEILING = 3
TABLE = 4
SEAT = 5
class ARPlaneAnchorPlaneClassificationStatus(betterproto.Enum):
UNKNOWN = 0
UNAVAILABLE = 1
UNDETERMINED = 2
KNOWN = 3
class CMCalibratedMagneticFieldCalibrationAccuracy(betterproto.Enum):
UNCALIBRATED = 0
LOW = 1
MEDIUM = 2
HIGH = 3
class ARMeshGeometryMeshClassification(betterproto.Enum):
NONE = 0
WALL = 1
FLOOR = 2
CEILING = 3
TABLE = 4
SEAT = 5
WINDOW = 6
DOOR = 7
class ObjectType(betterproto.Enum):
UNDEFINED_TYPE = 0
BOUNDING_BOX = 1
SKELETON = 2
MESH = 3
class ObjectMethod(betterproto.Enum):
UNKNOWN_METHOD = 0
ANNOTATION = 1
AUGMENTATION = 2
@dataclass
class AVCameraCalibrationData(betterproto.Message):
"""
Info about the camera characteristics used to capture images and depth
data.
"""
# 3x3 row-major matrix relating a camera's internal properties to an ideal
# pinhole-camera model.
intrinsic_matrix: List[float] = betterproto.float_field(1)
# The image dimensions to which the intrinsic_matrix values are relative.
intrinsic_matrix_reference_dimension_width: float = betterproto.float_field(2)
intrinsic_matrix_reference_dimension_height: float = betterproto.float_field(3)
# 3x4 row-major matrix relating a camera's position and orientation to a
# world or scene coordinate system. Consists of a unitless 3x3 rotation
# matrix (R) on the left and a translation (t) 3x1 vector on the right. The
# translation vector's units are millimeters. For example: |r1,1
# r2,1 r3,1 | t1| [R | t] = |r1,2 r2,2 r3,2 | t2| |r1,3 r2,3
# r3,3 | t3| is stored as [r11, r21, r31, t1, r12, r22, r32, t2, ...]
extrinsic_matrix: List[float] = betterproto.float_field(4)
# The size, in millimeters, of one image pixel.
pixel_size: float = betterproto.float_field(5)
# A list of floating-point values describing radial distortions imparted by
# the camera lens, for use in rectifying camera images.
lens_distortion_lookup_values: List[float] = betterproto.float_field(6)
# A list of floating-point values describing radial distortions for use in
# reapplying camera geometry to a rectified image.
inverse_lens_distortion_lookup_values: List[float] = betterproto.float_field(7)
# The offset of the distortion center of the camera lens from the top-left
# corner of the image.
lens_distortion_center_x: float = betterproto.float_field(8)
lens_distortion_center_y: float = betterproto.float_field(9)
@dataclass
class AVDepthData(betterproto.Message):
"""Container for depth data information."""
# PNG representation of the grayscale depth data map. See discussion about
# depth_data_map_original_minimum_value, below, for information about how to
# interpret the pixel values.
depth_data_map: bytes = betterproto.bytes_field(1)
# Pixel format type of the original captured depth data.
depth_data_type: str = betterproto.string_field(2)
depth_data_accuracy: AVDepthDataAccuracy = betterproto.enum_field(3)
# Indicates whether the depth_data_map contains temporally smoothed data.
depth_data_filtered: bool = betterproto.bool_field(4)
depth_data_quality: AVDepthDataQuality = betterproto.enum_field(5)
# Associated calibration data for the depth_data_map.
camera_calibration_data: AVCameraCalibrationData = betterproto.message_field(6)
# The original range of values expressed by the depth_data_map, before
# grayscale normalization. For example, if the minimum and maximum values
# indicate a range of [0.5, 2.2], and the depth_data_type value indicates it
# was a depth map, then white pixels (255, 255, 255) will map to 0.5 and
# black pixels (0, 0, 0) will map to 2.2 with the grayscale range linearly
# interpolated inbetween. Conversely, if the depth_data_type value indicates
# it was a disparity map, then white pixels will map to 2.2 and black pixels
# will map to 0.5.
depth_data_map_original_minimum_value: float = betterproto.float_field(7)
depth_data_map_original_maximum_value: float = betterproto.float_field(8)
# The width of the depth buffer map.
depth_data_map_width: int = betterproto.int32_field(9)
# The height of the depth buffer map.
depth_data_map_height: int = betterproto.int32_field(10)
# The row-major flattened array of the depth buffer map pixels. This will be
# either a float32 or float16 byte array, depending on 'depth_data_type'.
depth_data_map_raw_values: bytes = betterproto.bytes_field(11)
@dataclass
class ARLightEstimate(betterproto.Message):
"""
Estimated scene lighting information associated with a captured video
frame.
"""
# The estimated intensity, in lumens, of ambient light throughout the scene.
ambient_intensity: float = betterproto.double_field(1)
# The estimated color temperature, in degrees Kelvin, of ambient light
# throughout the scene.
ambient_color_temperature: float = betterproto.double_field(2)
# Data describing the estimated lighting environment in all directions.
# Second-level spherical harmonics in separate red, green, and blue data
# planes. Thus, this buffer contains 3 sets of 9 coefficients, or a total of
# 27 values.
spherical_harmonics_coefficients: List[float] = betterproto.float_field(3)
# A vector indicating the orientation of the strongest directional light
# source, normalized in the world-coordinate space.
primary_light_direction: ARLightEstimateDirectionVector = betterproto.message_field(4)
# The estimated intensity, in lumens, of the strongest directional light
# source in the scene.
primary_light_intensity: float = betterproto.float_field(5)
@dataclass
class ARLightEstimateDirectionVector(betterproto.Message):
x: float = betterproto.float_field(1)
y: float = betterproto.float_field(2)
z: float = betterproto.float_field(3)
@dataclass
class ARCamera(betterproto.Message):
"""
Information about the camera position and imaging characteristics for a
captured video frame.
"""
tracking_state: ARCameraTrackingState = betterproto.enum_field(1)
tracking_state_reason: ARCameraTrackingStateReason = betterproto.enum_field(2)
# 4x4 row-major matrix expressing position and orientation of the camera in
# world coordinate space.
transform: List[float] = betterproto.float_field(3)
euler_angles: ARCameraEulerAngles = betterproto.message_field(4)
# The width and height, in pixels, of the captured camera image.
image_resolution_width: int = betterproto.int32_field(5)
image_resolution_height: int = betterproto.int32_field(6)
# 3x3 row-major matrix that converts between the 2D camera plane and 3D world
# coordinate space.
intrinsics: List[float] = betterproto.float_field(7)
# 4x4 row-major transform matrix appropriate for rendering 3D content to
# match the image captured by the camera.
projection_matrix: List[float] = betterproto.float_field(8)
# 4x4 row-major transform matrix appropriate for converting from world-space
# to camera space. Relativized for the captured_image orientation (i.e.
# UILandscapeOrientationRight).
view_matrix: List[float] = betterproto.float_field(9)
@dataclass
class ARCameraEulerAngles(betterproto.Message):
"""
The orientation of the camera, expressed as roll, pitch, and yaw values.
"""
roll: float = betterproto.float_field(1)
pitch: float = betterproto.float_field(2)
yaw: float = betterproto.float_field(3)
@dataclass
class ARFaceGeometry(betterproto.Message):
"""Container for a 3D mesh describing face topology."""
vertices: List[ARFaceGeometryVertex] = betterproto.message_field(1)
# The number of elements in the vertices list.
vertex_count: int = betterproto.int32_field(2)
texture_coordinates: List[ARFaceGeometryTextureCoordinate] = betterproto.message_field(3)
# The number of elements in the texture_coordinates list.
texture_coordinate_count: int = betterproto.int32_field(4)
# Each integer value in this ordered list represents an index into the
# vertices and texture_coordinates lists. Each set of three indices
# identifies the vertices comprising a single triangle in the mesh. Each set
# of three indices forms a triangle, so the number of indices in the
# triangle_indices buffer is three times the triangle_count value.
triangle_indices: List[int] = betterproto.int32_field(5)
# The number of triangles described by the triangle_indices buffer.
triangle_count: int = betterproto.int32_field(6)
@dataclass
class ARFaceGeometryVertex(betterproto.Message):
"""
Each vertex represents a 3D point in the face mesh, in the face coordinate
space.
"""
x: float = betterproto.float_field(1)
y: float = betterproto.float_field(2)
z: float = betterproto.float_field(3)
@dataclass
class ARFaceGeometryTextureCoordinate(betterproto.Message):
"""
Each texture coordinate represents UV texture coordinates for the vertex at
the corresponding index in the vertices buffer.
"""
u: float = betterproto.float_field(1)
v: float = betterproto.float_field(2)
@dataclass
class ARBlendShapeMap(betterproto.Message):
"""
Contains a list of blend shape entries wherein each item maps a specific
blend shape location to its associated coefficient.
"""
entries: List[ARBlendShapeMapMapEntry] = betterproto.message_field(1)
@dataclass
class ARBlendShapeMapMapEntry(betterproto.Message):
# Identifier for the specific facial feature.
blend_shape_location: str = betterproto.string_field(1)
# Indicates the current position of the feature relative to its neutral
# configuration, ranging from 0.0 (neutral) to 1.0 (maximum movement).
blend_shape_coefficient: float = betterproto.float_field(2)
@dataclass
class ARFaceAnchor(betterproto.Message):
"""
Information about the pose, topology, and expression of a detected face.
"""
# A coarse triangle mesh representing the topology of the detected face.
geometry: ARFaceGeometry = betterproto.message_field(1)
# A map of named coefficients representing the detected facial expression in
# terms of the movement of specific facial features.
blend_shapes: ARBlendShapeMap = betterproto.message_field(2)
# 4x4 row-major matrix encoding the position, orientation, and scale of the
# anchor relative to the world coordinate space.
transform: List[float] = betterproto.float_field(3)
# Indicates whether the anchor's transform is valid. Frames that have a face
# anchor with this value set to NO should probably be ignored.
is_tracked: bool = betterproto.bool_field(4)
@dataclass
class ARPlaneGeometry(betterproto.Message):
"""Container for a 3D mesh."""
# A buffer of vertex positions for each point in the plane mesh.
vertices: List[ARPlaneGeometryVertex] = betterproto.message_field(1)
# The number of elements in the vertices buffer.
vertex_count: int = betterproto.int32_field(2)
# A buffer of texture coordinate values for each point in the plane mesh.
texture_coordinates: List[ARPlaneGeometryTextureCoordinate] = betterproto.message_field(3)
# The number of elements in the texture_coordinates buffer.
texture_coordinate_count: int = betterproto.int32_field(4)
# Each integer value in this ordered list represents an index into the
# vertices and texture_coordinates lists. Each set of three indices
# identifies the vertices comprising a single triangle in the mesh. Each set
# of three indices forms a triangle, so the number of indices in the
# triangle_indices buffer is three times the triangle_count value.
triangle_indices: List[int] = betterproto.int32_field(5)
# Each set of three indices forms a triangle, so the number of indices in the
# triangle_indices buffer is three times the triangle_count value.
triangle_count: int = betterproto.int32_field(6)
# Each value in this buffer represents the position of a vertex along the
# boundary polygon of the estimated plane. The owning plane anchor's
# transform matrix defines the coordinate system for these points.
boundary_vertices: List[ARPlaneGeometryVertex] = betterproto.message_field(7)
# The number of elements in the boundary_vertices buffer.
boundary_vertex_count: int = betterproto.int32_field(8)
@dataclass
class ARPlaneGeometryVertex(betterproto.Message):
x: float = betterproto.float_field(1)
y: float = betterproto.float_field(2)
z: float = betterproto.float_field(3)
@dataclass
class ARPlaneGeometryTextureCoordinate(betterproto.Message):
"""
Each texture coordinate represents UV texture coordinates for the vertex at
the corresponding index in the vertices buffer.
"""
u: float = betterproto.float_field(1)
v: float = betterproto.float_field(2)
@dataclass
class ARPlaneAnchor(betterproto.Message):
"""
Information about the position and orientation of a real-world flat
surface.
"""
# The ID of the plane.
identifier: str = betterproto.string_field(1)
# 4x4 row-major matrix encoding the position, orientation, and scale of the
# anchor relative to the world coordinate space.
transform: List[float] = betterproto.float_field(2)
# The general orientation of the detected plane with respect to gravity.
alignment: ARPlaneAnchorAlignment = betterproto.enum_field(3)
# A coarse triangle mesh representing the general shape of the detected
# plane.
geometry: ARPlaneGeometry = betterproto.message_field(4)
# The center point of the plane relative to its anchor position. Although the
# type of this property is a 3D vector, a plane anchor is always two-
# dimensional, and is always positioned in only the x and z directions
# relative to its transform position. (That is, the y-component of this
# vector is always zero.)
center: ARPlaneAnchorPlaneVector = betterproto.message_field(5)
# The estimated width and length of the detected plane.
extent: ARPlaneAnchorPlaneVector = betterproto.message_field(6)
# A Boolean value that indicates whether plane classification is available on
# the current device. On devices without plane classification support, all
# plane anchors report a classification value of NONE and a
# classification_status value of UNAVAILABLE.
classification_supported: bool = betterproto.bool_field(7)
# A general characterization of what kind of real-world surface the plane
# anchor represents.
classification: ARPlaneAnchorPlaneClassification = betterproto.enum_field(8)
# The current state of process for classifying the plane anchor. When this
# property's value is KNOWN, the classification property represents
# characterization of the real-world surface corresponding to the plane
# anchor.
classification_status: ARPlaneAnchorPlaneClassificationStatus = betterproto.enum_field(9)
@dataclass
class ARPlaneAnchorPlaneVector(betterproto.Message):
"""
Wrapper for a 3D point / vector within the plane. See extent and center
values for more information.
"""
x: float = betterproto.float_field(1)
y: float = betterproto.float_field(2)
z: float = betterproto.float_field(3)
@dataclass
class ARPointCloud(betterproto.Message):
"""A collection of points in the world coordinate space."""
# The number of points in the cloud.
count: int = betterproto.int32_field(1)
# The list of detected points.
point: List[ARPointCloudPoint] = betterproto.message_field(2)
# A list of unique identifiers corresponding to detected feature points. Each
# identifier in this list corresponds to the point at the same index in the
# points array.
identifier: List[int] = betterproto.int64_field(3)
@dataclass
class ARPointCloudPoint(betterproto.Message):
x: float = betterproto.float_field(1)
y: float = betterproto.float_field(2)
z: float = betterproto.float_field(3)
@dataclass
class CMVector(betterproto.Message):
"""A 3D vector"""
x: float = betterproto.double_field(1)
y: float = betterproto.double_field(2)
z: float = betterproto.double_field(3)
@dataclass
class CMCalibratedMagneticField(betterproto.Message):
"""
Represents calibrated magnetic field data and accuracy estimate of it.
"""
# Vector of magnetic field estimate.
field: CMVector = betterproto.message_field(1)
# Calibration accuracy of a magnetic field estimate.
calibration_accuracy: CMCalibratedMagneticFieldCalibrationAccuracy = betterproto.enum_field(2)
@dataclass
class CMDeviceMotion(betterproto.Message):
"""
A sample of device motion data. Encapsulates measurements of the attitude,
rotation rate, magnetic field, and acceleration of the device. Core Media
applies different algorithms of bias-reduction and stabilization to
rotation rate, magnetic field and acceleration values. For raw values check
correspondent fields in CMMotionManagerSnapshot object.
"""
# The device motion data object timestamp. May differ from the frame
# timestamp value since the data may be collected at higher rate.
timestamp: float = betterproto.double_field(1)
# The quaternion representing the device’s orientation relative to a known
# frame of reference at a point in time.
attitude_quaternion: CMDeviceMotionQuaternion = betterproto.message_field(2)
# The gravity acceleration vector expressed in the device's reference frame.
gravity: CMVector = betterproto.message_field(3)
# The acceleration that the user is giving to the device.
user_acceleration: CMVector = betterproto.message_field(4)
# Returns the magnetic field vector filtered with respect to the device bias.
magnetic_field: CMCalibratedMagneticField = betterproto.message_field(5)
# The rotation rate of the device adjusted by bias-removing Core Motion
# algorithms.
rotation_rate: CMVector = betterproto.message_field(6)
@dataclass
class CMDeviceMotionQuaternion(betterproto.Message):
x: float = betterproto.double_field(1)
y: float = betterproto.double_field(2)
z: float = betterproto.double_field(3)
w: float = betterproto.double_field(4)
@dataclass
class CMAccelerometerData(betterproto.Message):
"""A sample of raw accelerometer data."""
# The accelerometer data object timestamp. May differ from the frame
# timestamp value since the data may be collected at higher rate.
timestamp: float = betterproto.double_field(1)
# Raw acceleration measured by the accelerometer which effectively is a sum
# of gravity and user_acceleration of CMDeviceMotion object.
acceleration: CMVector = betterproto.message_field(2)
@dataclass
class CMGyroData(betterproto.Message):
"""A sample of raw gyroscope data."""
# The gyroscope data object timestamp. May differ from the frame timestamp
# value since the data may be collected at higher rate.
timestamp: float = betterproto.double_field(1)
# Raw rotation rate as measured by the gyroscope.
rotation_rate: CMVector = betterproto.message_field(2)
@dataclass
class CMMagnetometerData(betterproto.Message):
"""A sample of raw magnetometer data."""
# The magnetometer data object timestamp. May differ from the frame timestamp
# value since the data may be collected at higher rate.
timestamp: float = betterproto.double_field(1)
# Raw magnetic field measured by the magnetometer.
magnetic_field: CMVector = betterproto.message_field(2)
@dataclass
class CMMotionManagerSnapshot(betterproto.Message):
"""Contains most recent snapshots of device motion data"""
# Most recent samples of device motion data.
device_motion: List[CMDeviceMotion] = betterproto.message_field(1)
# Most recent samples of raw accelerometer data.
accelerometer_data: List[CMAccelerometerData] = betterproto.message_field(2)
# Most recent samples of raw gyroscope data.
gyro_data: List[CMGyroData] = betterproto.message_field(3)
# Most recent samples of raw magnetometer data.
magnetometer_data: List[CMMagnetometerData] = betterproto.message_field(4)
@dataclass
class ARFrame(betterproto.Message):
"""Video image and face position tracking information."""
# The timestamp for the frame.
timestamp: float = betterproto.double_field(1)
# The depth data associated with the frame. Not all frames have depth data.
depth_data: AVDepthData = betterproto.message_field(2)
# The depth data object timestamp associated with the frame. May differ from
# the frame timestamp value. Is only set when the frame has depth_data.
depth_data_timestamp: float = betterproto.double_field(3)
# Camera information associated with the frame.
camera: ARCamera = betterproto.message_field(4)
# Light information associated with the frame.
light_estimate: ARLightEstimate = betterproto.message_field(5)
# Face anchor information associated with the frame. Not all frames have an
# active face anchor.
face_anchor: ARFaceAnchor = betterproto.message_field(6)
# Plane anchors associated with the frame. Not all frames have a plane
# anchor. Plane anchors and face anchors are mutually exclusive.
plane_anchor: List[ARPlaneAnchor] = betterproto.message_field(7)
# The current intermediate results of the scene analysis used to perform
# world tracking.
raw_feature_points: ARPointCloud = betterproto.message_field(8)
# Snapshot of Core Motion CMMotionManager object containing most recent
# motion data associated with the frame. Since motion data capture rates can
# be higher than rates of AR capture, the entities of this object reflect all
# of the aggregated events which have occurred since the last ARFrame was
# recorded.
motion_manager_snapshot: CMMotionManagerSnapshot = betterproto.message_field(9)
@dataclass
class ARMeshGeometry(betterproto.Message):
"""Mesh geometry data stored in an array-based format."""
# The vertices of the mesh.
vertices: List[ARMeshGeometryVertex] = betterproto.message_field(1)
# The faces of the mesh.
faces: List[ARMeshGeometryFace] = betterproto.message_field(2)
# Rays that define which direction is outside for each face. Normals contain
# 'rays that define which direction is outside for each face', in practice
# the normals count is always identical to vertices count which looks like
# vertices normals and not faces normals.
normals: List[ARMeshGeometryVertex] = betterproto.message_field(3)
# Classification for each face in the mesh.
classification: List[ARMeshGeometryMeshClassification] = betterproto.enum_field(4)
@dataclass
class ARMeshGeometryVertex(betterproto.Message):
x: float = betterproto.float_field(1)
y: float = betterproto.float_field(2)
z: float = betterproto.float_field(3)
@dataclass
class ARMeshGeometryFace(betterproto.Message):
# / Indices of vertices defining the face from correspondent array of parent/
# message. A typical face is triangular.
vertex_indices: List[int] = betterproto.int32_field(1)
@dataclass
class ARMeshAnchor(betterproto.Message):
"""
A subdividision of the reconstructed, real-world scene surrounding the
user.
"""
# The ID of the mesh.
identifier: str = betterproto.string_field(1)
# 4x4 row-major matrix encoding the position, orientation, and scale of the
# anchor relative to the world coordinate space.
transform: List[float] = betterproto.float_field(2)
# 3D information about the mesh such as its shape and classifications.
geometry: ARMeshGeometry = betterproto.message_field(3)
@dataclass
class ARMeshData(betterproto.Message):
"""
Container object for mesh data of real-world scene surrounding the user.
Even though each ARFrame may have a set of ARMeshAnchors associated with
it, only a single frame's worth of mesh data is written separately at the
end of each recording due to concerns regarding latency and memory bloat.
"""
# The timestamp for the data.
timestamp: float = betterproto.double_field(1)
# Set of mesh anchors containing the mesh data.
mesh_anchor: List[ARMeshAnchor] = betterproto.message_field(2)
@dataclass
class KeyPoint(betterproto.Message):
# The position of the keypoint in the local coordinate system of the rigid
# object.
x: float = betterproto.float_field(1)
y: float = betterproto.float_field(2)
z: float = betterproto.float_field(3)
# Sphere around the keypoint, indicating annotator's confidence of the
# position in meters.
confidence_radius: float = betterproto.float_field(4)
# The name of the keypoint (e.g. legs, head, etc.). Does not have to be
# unique.
name: str = betterproto.string_field(5)
# Indicates whether the keypoint is hidden or not.
hidden: bool = betterproto.bool_field(6)
@dataclass
class Object(betterproto.Message):
# Unique object id through a sequence. There might be multiple objects of the
# same label in this sequence.
id: int = betterproto.int32_field(1)
# Describes what category an object is. E.g. object class, attribute,
# instance or person identity. This provides additional context for the
# object type.
category: str = betterproto.string_field(2)
type: ObjectType = betterproto.enum_field(3)
# 3x3 row-major rotation matrix describing the orientation of the rigid
# object's frame of reference in the world-coordinate system.
rotation: List[float] = betterproto.float_field(4)
# 3x1 vector describing the translation of the rigid object's frame of
# reference in the world-coordinate system in meters.
translation: List[float] = betterproto.float_field(5)
# 3x1 vector describing the scale of the rigid object's frame of reference in
# the world-coordinate system in meters.
scale: List[float] = betterproto.float_field(6)
# List of all the key points associated with this object in the object
# coordinate system. The first keypoint is always the object's frame of
# reference, e.g. the centroid of the box. E.g. bounding box with its center
# as frame of reference, the 9 keypoints : {0., 0., 0.}, {-.5, -.5, -.5},
# {-.5, -.5, +.5}, {-.5, +.5, -.5}, {-.5, +.5, +.5}, {+.5, -.5, -.5}, {+.5,
# -.5, +.5}, {+.5, +.5, -.5}, {+.5, +.5, +.5} To get the bounding box in the
# world-coordinate system, we first scale the box then transform the scaled
# box. For example, bounding box in the world coordinate system is rotation *
# scale * keypoints + translation
keypoints: List[KeyPoint] = betterproto.message_field(7)
method: ObjectMethod = betterproto.enum_field(8)
@dataclass
class Edge(betterproto.Message):
"""The edge connecting two keypoints together"""
# keypoint id of the edge's source
source: int = betterproto.int32_field(1)
# keypoint id of the edge's sink
sink: int = betterproto.int32_field(2)
@dataclass
class Skeleton(betterproto.Message):
"""
The skeleton template for different objects (e.g. humans, chairs, hands,
etc) The annotation tool reads the skeleton template dictionary.
"""
# The origin keypoint in the object coordinate system. (i.e. Point 0, 0, 0)
reference_keypoint: int = betterproto.int32_field(1)
# The skeleton's category (e.g. human, chair, hand.). Should be unique in the
# dictionary.
category: str = betterproto.string_field(2)
# Initialization value for all the keypoints in the skeleton in the object's
# local coordinate system. Pursuit will transform these points using object's
# transformation to get the keypoint in the world-coordinate.
keypoints: List[KeyPoint] = betterproto.message_field(3)
# List of edges connecting keypoints
edges: List[Edge] = betterproto.message_field(4)
@dataclass
class Skeletons(betterproto.Message):
"""
The list of all the modeled skeletons in our library. These models can be
objects (chairs, desks, etc), humans (full pose, hands, faces, etc), or
box. We can have multiple skeletons in the same file.
"""
object: List[Skeleton] = betterproto.message_field(1)
@dataclass
class NormalizedPoint2D(betterproto.Message):
"""Projection of a 3D point on an image, and its metric depth."""
# x-y position of the 2d keypoint in the image coordinate system. u,v \in [0,
# 1], where top left corner is (0, 0) and the bottom-right corner is (1, 1).
x: float = betterproto.float_field(1)
y: float = betterproto.float_field(2)
# The depth of the point in the camera coordinate system (in meters).
depth: float = betterproto.float_field(3)
@dataclass
class Point3D(betterproto.Message):
"""
The 3D point in the camera coordinate system, the scales are in meters.
"""
x: float = betterproto.float_field(1)
y: float = betterproto.float_field(2)
z: float = betterproto.float_field(3)
@dataclass
class AnnotatedKeyPoint(betterproto.Message):
id: int = betterproto.int32_field(1)
point_3d: Point3D = betterproto.message_field(2)
point_2d: NormalizedPoint2D = betterproto.message_field(3)
@dataclass
class ObjectAnnotation(betterproto.Message):
# Reference to the object identifier in ObjectInstance.
object_id: int = betterproto.int32_field(1)
# For each objects, list all the annotated keypoints here. E.g. for bounding-
# boxes, we have 8 keypoints, hands = 21 keypoints, etc. These normalized
# points are the projection of the Object's 3D keypoint on the current
# frame's camera poses.
keypoints: List[AnnotatedKeyPoint] = betterproto.message_field(2)
# Visibiity of this annotation in a frame.
visibility: float = betterproto.float_field(3)
@dataclass
class FrameAnnotation(betterproto.Message):
# Unique frame id, corresponds to images.
frame_id: int = betterproto.int32_field(1)
# List of the annotated objects in this frame. Depending on how many object
# are observable in this frame, we might have non or as much as
# sequence.objects_size() annotations.
annotations: List[ObjectAnnotation] = betterproto.message_field(2)
# Information about the camera transformation (in the world coordinate) and
# imaging characteristics for a captured video frame.
camera: ARCamera = betterproto.message_field(3)
# The timestamp for the frame.
timestamp: float = betterproto.double_field(4)
# Plane center and normal in camera frame.
plane_center: List[float] = betterproto.float_field(5)
plane_normal: List[float] = betterproto.float_field(6)
@dataclass
class Sequence(betterproto.Message):
"""
The sequence protocol contains the annotation data for the entire video clip.
"""
# List of all the annotated 3D objects in this sequence in the world
# Coordinate system. Given the camera poses of each frame (also in the world-
# coordinate) these objects bounding boxes can be projected to each frame to
# get the per-frame annotation (i.e. image_annotation below).
objects: List[Object] = betterproto.message_field(1)
# List of annotated data per each frame in sequence + frame information.
frame_annotations: List[FrameAnnotation] = betterproto.message_field(2)
examples/python/objectron/pyproject.toml
+19
View File
@@ -0,0 +1,19 @@
[project]
name = "objectron"
version = "0.1.0"
readme = "README.md"
dependencies = [
"betterproto[compiler]",
"numpy",
"opencv-python>4.6",
"requests>=2.31,<3",
"rerun-sdk",
"scipy",
]
[project.scripts]
objectron = "objectron.__main__:main"
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"