tracer-cloud--opensre/docs/technology/end-to-end.mdx

---
title: 'How Tracer works, end to end'
sidebarTitle: 'How Tracer works'
description: 'A shared execution signal for analysis and optimization'
---

Tracer is built to make execution behavior visible in compute-intensive environments, without changing workloads or relying on what applications choose to report.

At a high level, Tracer works in three layers:

- **Tracer/collect**: an open-source eBPF agent that gathers execution signals from the host-layer
- **Tracer/datalake**: a shared execution view across pipelines and environments
- **Tracer/tune and Tracer/sweep**: use that signal to solve different problems

<Frame>
  <img src="/images/tracersimplified.webp" alt="Tracer simplified architecture: collect, datalake, tune and sweep" />
</Frame>

This page explains the architecture. Each product page goes deeper on its specific behavior.

## What Tracer is made of

Tracer consists of three components with distinct responsibilities:

- **Tracer/collect** gathers execution signals directly from the operating system
- **Tracer/tune** uses those signals to analyze and optimize pipeline performance
- **Tracer/sweep** uses the same signals to uncover systemwide cloud waste

Tracer/collect is the foundation. Tracer/tune and Tracer/sweep are built on top of the execution signal it produces.

<Frame>
  <img src="/images/howTracerWorks.webp" alt="How Tracer works end to end: from kernel-level signals to analysis and optimization" />
</Frame>

## Architecture at a glance

Tracer's data flow can be understood in four stages:

<Steps>
  <Step title="Attach" titleSize="h3">
    Tracer/collect attaches non-intrusively to running processes and containers on a Linux host using eBPF, a Linux kernel technology for safe, low-overhead instrumentation. No code changes, container restarts, or application modifications are required.
  </Step>
  <Step title="Collect" titleSize="h3">
    Execution events are captured at the kernel boundary, including CPU scheduling, memory activity, disk and network I/O, and process lifecycle events. Only relevant signals are selected through intelligent filtering rules.
  </Step>
  <Step title="Correlate" titleSize="h3">
    Low-level events are mapped to higher-level execution context such as containers, tools, tasks, runs, and pipelines. This mapping uses kernel-native identifiers like PIDs, namespaces, and cgroups.
  </Step>
  <Step title="Stream" titleSize="h3">
    Structured telemetry is batched and sent securely to Tracer's backend, where it becomes available for analysis, visualization, and downstream products. Data is buffered locally and retried until successfully delivered.
  </Step>
</Steps>

This pipeline is continuous and designed to operate safely in production cloud compute environments.

## The execution signal (single source of truth)

Tracer's execution signal is a structured representation of what actually ran on the system.

It includes:

- CPU usage and scheduling behavior
- Memory allocation and pressure
- Disk and network I/O activity
- Process lifecycles and relationships
- Container and host context

<Note>
**It explicitly does not include:**
- Application payloads or scientific input/output data
- Source code, function calls, or language-level execution traces
- Application- or domain-specific interpretation of what a command does
</Note>

The execution signal is derived from kernel-level observation via eBPF, without application instrumentation or code changes. It serves as the shared input for both Tracer/tune and Tracer/sweep.

## How correlation works

Raw kernel events are not useful on their own. Tracer/collect correlates them into meaningful execution context.

At a high level:

- Kernel events are associated with processes
- Processes are grouped by containers and cgroups
- Containers and processes are mapped to tools, tasks, runs, and pipelines

This correlation allows Tracer to answer questions such as:

- Which tool generated this I/O?
- Which task was idle during this period?
- Which pipeline run consumed these resources?

All correlation is derived from operating system identifiers and execution context, not from workflow-specific integrations.

## Where Tracer/tune fits

Tracer/tune focuses on pipelines that already work, but are slow or inefficient.

Using the execution signal, Tracer/tune:

- Visualizes actual resource usage at the task and process level
- Identifies underutilization, contention, and bottlenecks
- Distinguishes compute-bound, memory-bound, and I/O-bound stages
- Produces evidence-based recommendations for right-sizing and optimization

Tracer/tune answers: **"How do we make this pipeline faster and cheaper?"**

<Card href="/technology/tracer-tune">
  <span style={{ fontSize: '1.25rem', fontWeight: '500' }}>
    <span style={{ background: 'linear-gradient(135deg, #FCFCFC, #C4C4C4)', WebkitBackgroundClip: 'text', WebkitTextFillColor: 'transparent', backgroundClip: 'text' }}>Tracer/</span><span style={{ background: 'linear-gradient(135deg, #38BDA4, #76E9D3)', WebkitBackgroundClip: 'text', WebkitTextFillColor: 'transparent', backgroundClip: 'text' }}>tune</span>
  </span>
  <br />
  Learn more about pipeline performance optimization
</Card>

## Where Tracer/sweep fits

Tracer/sweep focuses on systemwide cloud efficiency.

Using the same execution signal, Tracer/sweep:

- Scans cloud compute based on real execution activity
- Identifies idle time, unused capacity, and hidden inefficiencies
- Surfaces waste that does not appear in billing reports or dashboards
- Avoids predictive shutdown heuristics by relying on observed behavior

Tracer/sweep answers: **"Where are we wasting cloud spend right now?"**

<Card href="/technology/tracer-sweep">
  <span style={{ fontSize: '1.25rem', fontWeight: '500' }}>
    <span style={{ background: 'linear-gradient(135deg, #FCFCFC, #C4C4C4)', WebkitBackgroundClip: 'text', WebkitTextFillColor: 'transparent', backgroundClip: 'text' }}>Tracer/</span><span style={{ background: 'linear-gradient(135deg, #4436BD, #5646E2)', WebkitBackgroundClip: 'text', WebkitTextFillColor: 'transparent', backgroundClip: 'text' }}>sweep</span>
  </span>
  <br />
  Learn more about cloud waste detection
</Card>

## Choose your path

Depending on your goal, you can go deeper in different directions:

<CardGroup cols={3}>
  <Card href="/technology/tracer-collect">
    <span style={{ fontSize: '1.25rem', fontWeight: '500' }}>
      <span style={{ background: 'linear-gradient(135deg, #FCFCFC, #C4C4C4)', WebkitBackgroundClip: 'text', WebkitTextFillColor: 'transparent', backgroundClip: 'text' }}>Tracer/</span><span style={{ background: 'linear-gradient(135deg, #FB68E1, #953E96)', WebkitBackgroundClip: 'text', WebkitTextFillColor: 'transparent', backgroundClip: 'text' }}>collect</span>
    </span>
    <br />
    Learn how execution signals are captured safely and efficiently at the kernel level.
  </Card>
  <Card href="/technology/tracer-tune">
    <span style={{ fontSize: '1.25rem', fontWeight: '500' }}>
      <span style={{ background: 'linear-gradient(135deg, #FCFCFC, #C4C4C4)', WebkitBackgroundClip: 'text', WebkitTextFillColor: 'transparent', backgroundClip: 'text' }}>Tracer/</span><span style={{ background: 'linear-gradient(135deg, #38BDA4, #76E9D3)', WebkitBackgroundClip: 'text', WebkitTextFillColor: 'transparent', backgroundClip: 'text' }}>tune</span>
    </span>
    <br />
    Learn how Tracer turns execution data into pipeline performance insights and recommendations.
  </Card>
  <Card href="/technology/tracer-sweep">
    <span style={{ fontSize: '1.25rem', fontWeight: '500' }}>
      <span style={{ background: 'linear-gradient(135deg, #FCFCFC, #C4C4C4)', WebkitBackgroundClip: 'text', WebkitTextFillColor: 'transparent', backgroundClip: 'text' }}>Tracer/</span><span style={{ background: 'linear-gradient(135deg, #4436BD, #5646E2)', WebkitBackgroundClip: 'text', WebkitTextFillColor: 'transparent', backgroundClip: 'text' }}>sweep</span>
    </span>
    <br />
    Learn how Tracer uncovers cloud waste using real activity patterns.
  </Card>
</CardGroup>

<div style={{ height: '50vh' }}></div>