# Rerun architecture

This document describes the technical architecture of Rerun.

## See also

-   [`BUILD.md`](BUILD.md)
-   [`TESTING.md`](TESTING.md)
-   [`rerun_py/README.md`](rerun_py/README.md) - build instructions for Python SDK
-   [`CODE_OF_CONDUCT.md`](CODE_OF_CONDUCT.md)
-   [`CODE_STYLE.md`](CODE_STYLE.md)
-   [`CONTRIBUTING.md`](CONTRIBUTING.md)
-   [`RELEASES.md`](RELEASES.md)

## The major components

### Logging APIs

It all starts with logging. You can log rich data (point clouds, images, etc.) with either our Python SDK or our Rust SDK.

The logging SDK:s encodes the data using Apache Arrow (see more below).

The logging data can be written to disk as `.rrd` files, or transmitted over gRPC to either a Rerun Viewer or a Rerun Server.

### Rerun Viewer

The Rerun Viewer is where log data is visualized. It is usually run as a native app, but can also be compiled to WebAssembly (Wasm) and run in a browser.

#### Native Viewer

The easiest way to launch the Viewer is directly from the logging API with `rr.init("rerun_example_app", spawn=True)`. However, the standalone Viewer can also be run from the command line, for example to view an `.rrd` file: `rerun mydata.rrd`.

#### Web viewer

You can try running the Viewer in a browser using `rr.serve()` in Python, or using `rerun --web-viewer mydata.rrd`.

The web viewer consists of just a few small files - a thin `.html`, a `.wasm` blob, and an auto-generated `.js` bridge for the wasm. These files are served using the [`re_web_viewer_server`](https://github.com/rerun-io/rerun/tree/latest/crates/viewer/re_web_viewer_server) crate.

The web viewer can load `.rrd` files (just drag-drop them into the browser), or read logging data streamed over gRPC.

### `.rrd` files

`.rrd` ("**R**e**r**un **D**ata") is just a bunch of log messages appended one after the other to a file.

NOTE: `.rrd` files do not yet offer full backwards or forwards compatibility. However, the current version of Rerun will always be able to open `.rrd` files generated by the *previous* version. This is not guaranteed to work for files generated with older versions.

## Technologies we use

### Apache Arrow

[Apache Arrow](https://arrow.apache.org/) is a language-independent columnar memory format for arbitrary data. We use it to encode the log data when transmitting it over the network or storing it in an `.rrd` file. We also use it in our in-RAM data store, [`re_chunk_store`](crates/store/re_chunk_store/README.md).

In Rust, we use the [`arrow` crate](https://crates.io/crates/arrow).

### `wgpu`

The Rerun Viewer uses the [`wgpu`](https://github.com/gfx-rs/wgpu) graphics API. It provides a high-performance abstraction over Vulkan, Metal, D3D12, D3D11, OpenGLES, WebGL and [WebGPU](https://en.wikipedia.org/wiki/WebGPU). This lets us write the same code graphics code for native as for web.

On web builds, we use WebGPU when available on the Web, but automatically fall back to a WebGL based emulation layer (with a more limited feature set).

We have written our own high-level rendering crate on top of `wgpu`, called [`re_renderer`](crates/viewer/re_renderer/README.md).

### `egui`

The GUI in the Rerun Viewer is using [`egui`](https://www.egui.rs/), a cross-platform, [immediate mode GUI](https://github.com/emilk/egui#why-immediate-mode).

We use [`eframe`](https://github.com/emilk/egui/tree/master/crates/eframe), the egui framework, to run `egui` on both native and web.

### Wasm

Wasm (short for [WebAssembly](https://webassembly.org/)) is a binary instruction format supported by all major browser.
The Rerun Viewer can be compiled to Wasm and run in a browser.

Threading support in Wasm is nascent, so care must be taken that we don't spawn any threads when compiling for `wasm32`.

Wasm has no access to the host system, except via JS calls (something that may change once [WASI](https://wasi.dev/) rolls out), so when compiling for `wasm32` you can NOT use the Rust standard library to:

-   Access files
-   Read environment variables
-   Get the current time (use [`instant`](https://crates.io/crates/instant) instead)
-   Use networking (use [`ehttp`](https://github.com/emilk/ehttp), [`reqwest`](https://github.com/seanmonstar/reqwest), or [`ewebsock`](https://github.com/rerun-io/ewebsock) instead)
-   etc

## Immediate mode

The Rerun Viewer uses an [immediate mode GUI](https://github.com/emilk/egui#why-immediate-mode), [`egui`](https://www.egui.rs/). This means that each frame the entire GUI is being laid out from scratch.

In fact, the whole of the Rerun Viewer is written in an immediate mode style. Each rendered frame it will query the in-RAM data store, massage the results, and feed it to the renderer.

The advantage of immediate mode is that is removes all state management. There is no callbacks that are called when some state has already changed, and the state of the blueprint is always in sync with what you see on screen.

Immediate mode is also a forcing function, forcing us to relentlessly optimize our code.
This leads to a very responsive GUI, where there is no "hiccups" when switching data source or doing time scrubbing.

Of course, this will only take us so far. In the future we plan on caching queries and work submitted to the renderer so that we don't perform unnecessary work each frame. We also plan on doing larger operation in background threads. This will be necessary in order to support viewing large datasets, e.g. several million points. The plan is still to do so within an immediate mode framework, retaining most of the advantages of stateless code.

## Crates

Here is an overview of the crates included in the project:

<picture>
  <img src="https://static.rerun.io/architecture_diagram/ab6062b1b48f8edcd64922547aad6b493616d3f4/full.png" alt="">
  <source media="(max-width: 480px)" srcset="https://static.rerun.io/architecture_diagram/ab6062b1b48f8edcd64922547aad6b493616d3f4/480w.png">
  <source media="(max-width: 768px)" srcset="https://static.rerun.io/architecture_diagram/ab6062b1b48f8edcd64922547aad6b493616d3f4/768w.png">
  <source media="(max-width: 1024px)" srcset="https://static.rerun.io/architecture_diagram/ab6062b1b48f8edcd64922547aad6b493616d3f4/1024w.png">
  <source media="(max-width: 1200px)" srcset="https://static.rerun.io/architecture_diagram/ab6062b1b48f8edcd64922547aad6b493616d3f4/1200w.png">
</picture>

<!-- !!! IMPORTANT!!!

This image must be updated each time a crate is added/removed/updated.

FigJam document: https://www.figma.com/file/Umob8ztK1HmYKLUMSq8aPb/Crates-org

Update instructions:
1) Update the FigJam document
2) Select all -> right-click -> Copy as PNG
3) `pixi run upload-image --name architecture_diagram`
4) Copy/paste the resulting HTML
-->

### SDK/CLI/Wasm top-level crates

| Crate     | Description                          |
| --------- | ------------------------------------ |
| rerun-cli | Rerun native CLI binary crate        |
| Rerun     | Rerun Rust SDK and Viewer shim crate |
| rerun_c   | Rerun C SDK                          |
| rerun_py  | Rerun Python SDK                     |
| re_sdk    | Rerun logging SDK                    |

### Viewer crates

##### UI crates

| Crate                  | Description                                                                                                |
| ---------------------- |------------------------------------------------------------------------------------------------------------|
| re_arrow_ui            | Show arrow data in a tree of rerun list_items and format arrow with syntax highlighting.                   |
| re_blueprint_tree      | The UI for the blueprint tree in the left panel.                                                           |
| re_chunk_store_ui      | A chunk store browser UI.                                                                                  |
| re_component_fallbacks | Provides default fallbacks for Rerun components in the fallback provider registry.                         |
| re_component_ui        | Provides UI editors for Rerun component data for registration with the Rerun Viewer component UI registry. |
| re_dataframe_ui        | Rich table widget over DataFusion.                                                                         |
| re_gamepad             | Gamepad input handling for the Rerun viewer.                                                               |
| re_memory_view         | Flamegraph visualization for memory usage trees                                                            |
| re_plot                | Plot ui components shared between time series and bar chart view.                                          |
| re_recording_panel     | The UI for the recording panel.                                                                            |
| re_redap_browser       | The UI and communication to implement the in-viewer redap server browser.                                  |
| re_selection_panel     | The UI for the selection panel.                                                                            |
| re_time_panel          | The time panel of the Rerun Viewer, allowing to control the displayed timeline & time.                     |
| re_time_ruler          | Time ruler widget shared by the time panel and time-aware views.                                           |
| re_view                | Types & utilities for defining View classes and communicating with the Viewport.                           |
| re_view_bar_chart      | A View that shows a single bar chart.                                                                      |
| re_view_dataframe      | A View that shows the data contained in entities in a table.                                               |
| re_view_graph          | A View that shows a graph (node-link diagram).                                                             |
| re_view_map            | A View that shows geospatial data on a map.                                                                |
| re_view_spatial        | Views that show entities in a 2D or 3D spatial relationship.                                               |
| re_view_state_timeline | A view that shows state transitions as horizontal lanes over time.                                         |
| re_view_tensor         | A View dedicated to visualizing tensors with arbitrary dimensionality.                                     |
| re_view_text_document  | A simple View that shows a single text box.                                                                |
| re_view_text_log       | A View that shows text entries in a table and scrolls with the active time.                                |
| re_view_time_series    | A View that shows plots over Rerun timelines.                                                              |
| re_viewer              | The Rerun Viewer                                                                                           |
| re_viewer_mcp          | MCP server that allows agents to interact with the viewer.                                                 |
| re_viewport            | The central viewport panel of the Rerun viewer.                                                            |

##### UI support crates

| Crate                 | Description                                                          |
| --------------------- | -------------------------------------------------------------------- |
| re_context_menu       | Support crate for context menu and actions.                          |
| re_data_ui            | Provides UI elements for Rerun component data for the Rerun Viewer.  |
| re_renderer           | A wgpu-based renderer for all your visualization needs.              |
| re_renderer_examples  | Examples for re_renderer                                             |
| re_ui                 | Rerun GUI theme and helpers, built around egui                       |
| re_viewer_context     | Rerun Viewer state that is shared with the viewer's code components. |
| re_viewport_blueprint | The data model description of the viewport panel.                    |

### Application-level store

| Crate           | Description                                                     |
| --------------- | --------------------------------------------------------------- |
| re_dataframe    | The Rerun public data APIs.                                     |
| re_datafusion   | DataFusion interfaces to Rerun gRPC queries                     |
| re_entity_db    | In-memory storage of Rerun entities                             |
| re_log_encoding | Helpers for encoding and transporting Rerun log messages        |
| re_protos       | Rerun remote store gRPC API types                               |
| re_query        | Querying data in the re_chunk_store                             |
| re_server       | In-memory data server                                           |
| re_tf           | Dealing with spatial transforms & transform frames              |
| re_sdk_types    | The built-in Rerun data types, component types, and archetypes. |

### Low-level store

| Crate          | Description                                                                                   |
| -------------- | --------------------------------------------------------------------------------------------- |
| re_chunk       | A chunk of Rerun data, encoded using Arrow. Used for logging, transport, storage and compute. |
| re_chunk_store | An in-memory time series database for Rerun log data, based on Apache Arrow.                  |
| re_log_types   | The basic building blocks of the Rerun data types and tables.                                 |
| re_sorbet      | Rerun arrow metadata and record batch definitions.                                            |
| re_types_core  | The core traits and types that power Rerun's data model.                                      |

### Data flow

| Crate                | Description                                                       |
| -------------------- | ----------------------------------------------------------------- |
| re_redap_client      | Official client for the Rerun Data Protocol                       |
| re_redap_tests       | Official test suite for the Rerun Data Protocol                   |
| re_importer          | Handles importing of Rerun data from file using importer plugins  |
| re_parquet           | Parquet file loading with configurable column grouping            |
| re_mp4_reader        | Mp4 video file loading into Rerun chunks                          |
| re_data_source       | Handles loading of Rerun data from different sources              |
| re_grpc_client       | Client for the legacy StoreHub API                                |
| re_grpc_server       | Server for the legacy StoreHub API                                |
| re_web_viewer_server | Serves the Rerun web viewer (Wasm and HTML) over HTTP             |

### Build support

| Crate             | Description                                                      |
| ----------------- | ---------------------------------------------------------------- |
| re_build_info     | Information about the build. Use together with re_build_tools    |
| re_build_tools    | build.rs helpers for generating build info                       |
| re_dev_tools      | Various tools for Rerun development. Each tool has a subcommand. |
| re_protos_builder | Generates code for Rerun remote store gRPC API                   |
| re_types_builder  | Generates code for Rerun's SDKs from flatbuffers definitions.    |

### Utilities

| Crate                | Description                                                                                   |
| -------------------- | --------------------------------------------------------------------------------------------- |
| re_analytics         | Rerun's analytics SDK                                                                         |
| re_arrow_util        | Helpers for working with arrow                                                                |
| re_auth              | Authentication and authorization helpers                                                      |
| re_backoff           | Simple backoff logic used for retries.                                                        |
| re_byte_size         | Calculate the heap-allocated size of values at runtime                                        |
| re_byte_size_derive  | Derive macro for `re_byte_size::SizeBytes`                                                    |
| re_capabilities      | Capability tokens                                                                             |
| re_case              | Case conversions, the way Rerun likes them                                                    |
| re_crash_handler     | Detect panics and signals, logging them and optionally sending them to analytics.             |
| re_error             | Helpers for handling errors.                                                                  |
| re_format            | Miscellaneous tools to format and parse numbers, durations, etc.                              |
| re_grpc_headers      | Rerun gRPC header conventions (`x-rerun-*` consts, version interceptor, tower layers).        |
| re_lenses            | A collection of lenses for mapping component data to Rerun semantic types.                    |
| re_lenses_core       | Core lens types and composable Arrow array transformations                                    |
| re_log               | Helpers for setting up and doing text logging in the Rerun crates.                            |
| re_log_channel       | An in-memory channel of Rerun data messages                                                   |
| re_mcap              | Convert MCAP into Rerun-compatible data.                                                      |
| re_memory            | Run-time memory tracking and profiling.                                                       |
| re_mutex             | A wrapper around `parking_lot::Mutex` useful for debugging deadlocks.                         |
| re_perf_telemetry    | In and out of process performance profiling utilities for Rerun & Redap                       |
| re_quota_channel     | A mpsc channel that applies backpressure based on byte size                                   |
| re_ros_msg           | Parsing and deserializing ROS   messages                                                      |
| re_rvl               | Codecs for depth compression including RVL and PNG.                                           |
| re_span              | An integer range that always has a non-negative length                                        |
| re_string_interner   | Yet another string interning library                                                          |
| re_test_mocks        | In-process server doubles (`MockOtlpCollector`, `MockPostHog`) for OTel/PostHog test capture. |
| re_tracing           | Helpers for tracing/spans/flamegraphs and such.                                               |
| re_tuid              | 128-bit Time-based Unique Identifier                                                          |
| re_uri               | Parsing and constructing of Rerun URIs                                                        |
| re_video             | Video decoding library                                                                        |

### Test crates

| Crate            | Description                                                   |
| ---------------- | ------------------------------------------------------------- |
| re_test_context  | Provides a test context that builds on `re_viewer_context.    |
| re_test_viewport | Extends the `re_test_context` with viewport-related features. |

### Deprecated crates

| Crate    | Description                |
| -------- | -------------------------- |
| re_types | Old name of `re_sdk_types` |

### Dependencies and docs

In order to get a dependency graph for our in-house crates and their docs, we recommend you run:

```
cargo install cargo-depgraph
cargo depgraph --all-deps --workspace-only --all-features --dedup-transitive-deps | dot -Tpng > /tmp/rerun-deps.png
open /tmp/rerun-deps.png
```

and:

```
cargo doc --no-deps --open
```

and then browse through the `re_*` crates.
