--- title: 'Tracer/tune' sidebarTitle: 'Tracer/tune' description: 'Turn "working" pipelines into fast, efficient pipelines' --- Tracer/tune is where execution visibility becomes optimization. It answers what actually happened at runtime and turns that understanding into concrete recommendations to improve pipeline performance, stability, and cost efficiency. Tracer/tune is built on execution signals captured by Tracer/collect. It does not change pipeline code, rewrite workflows, or rely on heuristics. All recommendations are based on observed execution behavior. ## What Tracer/tune does Tracer/tune analyzes how pipelines actually ran and translates that behavior into actionable guidance. Specifically, it: - Reconstructs execution at the level of pipelines, runs, steps, tools, and subprocesses - Visualizes real CPU, memory, disk, and network usage over time - Identifies bottlenecks, idle execution, and over-allocation - Produces right-sizing and optimization recommendations grounded in runtime data Tracer/tune focuses on execution reality, not configuration intent. Runtime view showing pipeline execution organized by pipeline, run, step, and tool ## What Tracer/tune is optimized for Tracer/tune is designed for teams with pipelines that fail or already run successfully but are inefficient, unstable, or expensive. Understand patterns across many runs, not single outliers Pinpoint what actually limits progress Align resources with observed usage Reduce retries, stalls, and intermittent failures Detect performance drift over time These problems are difficult to solve with logs, dashboards, or orchestration metadata alone. ## How Tracer/tune produces recommendations Tracer/tune operates entirely on execution signals derived from the host-layer. ### Inputs Tracer/tune analyzes: - CPU utilization and scheduling behavior - Memory usage, peak memory, and pressure - Disk and network I/O throughput and wait time - Idle execution and blocked subprocesses - Variance in resource usage across runs and steps These signals come from kernel-level telemetry and reflect what actually happened during execution. Kernel-level telemetry showing CPU, memory, disk I/O, and network activity metrics ### Outputs Based on these observations, Tracer/tune produces recommendations such as: - Lowering CPU or memory requests for underutilized steps - Increasing peak memory to prevent OOM retries - Changing storage type or data locality for I/O-bound stages - Selecting more appropriate instance or node families - Highlighting steps that stall, make no forward progress, or run abnormally slow Recommendations are advisory, explainable, and based only on observed behavior. ## What you see in the Tracer UI Tracer/tune is driven by a shared execution view in the Tracer UI. You can: - Follow pipeline runs in real time, step by step - See which tools and subprocesses are active, queued, or stalled - Inspect resource usage over time for each step or tool - Compare behavior across runs to identify regressions or improvements - Understand how work is distributed across nodes and instances Root-cause insights correlating slowdowns and failures with resource behavior Automatic logging showing structured execution timelines from kernel-level signals Tracer/tune generates complete, structured execution timelines directly from kernel-level signals, even for tools that produce minimal logs, logs that disappear after a failure, or no logs at all. Because logging is derived from the operating system rather than the application, you always get complete runtime information without instrumentation, wrappers, or re-running pipelines. This view is reconstructed directly from execution signals and does not depend on workflow metadata or application logs. ## Examples These examples are framework-agnostic and apply across workflow engines and environments. ### CPU underutilization A step requests many cores but consistently uses only a small fraction. → Tracer/tune recommends lowering CPU allocation without affecting runtime. Kernel-level telemetry using eBPF for low-level performance observation ### High I/O wait A task spends most of its time blocked on disk or network I/O. → Tracer/tune recommends storage or locality changes, not additional cores. Failure signals showing OOM kills, stalled tools, and I/O wait issues ### Memory spikes and retries A step occasionally exceeds memory limits and retries. → Tracer/tune recommends right-sizing peak memory to stabilize execution. In each case, the recommendation is tied directly to observed runtime behavior. ## Cost-aware optimization Tracer/tune links execution behavior to actual cloud cost. It: - Breaks down usage and cost by pipeline, run, step, tool, and instance - Uses cloud-provider billing metrics for accurate cost attribution - Highlights over-provisioned resources that drive unnecessary spend - Supports instance rightsizing and instance family recommendations Cost and usage tracking broken down by pipeline, run, step, tool, and instance Instance rightsizing recommendations based on real resource usage Cost optimization is derived from execution data, not estimates or tagging. ## What Tracer/tune does not replace **Tracer/tune is not:** - A workflow orchestrator or scheduler - A language-level or function-level profiler Tracer/tune provides process-level execution truth. For code-level optimization, it complements traditional profilers rather than replacing them. ## Requirements Tracer/tune requires Tracer/collect to be installed and running. It supports: - AWS Batch and other Linux-based cloud compute - On-prem and hybrid HPC environments - Containerized and non-containerized workloads - Any workflow engine, scheduler, language, or binary supported by Tracer/collect No code changes, instrumentation, or tagging are required. Once installed, your next run is visible automatically. See [Quickstart](/quickstart). ## Summary Tracer/tune turns execution visibility into optimization. By analyzing what actually happened at runtime, it helps teams make informed decisions about resource allocation, performance tuning, and cost control, without rewriting pipelines or changing how they work. Tracer/collect
Learn how execution data is captured Tracer/sweep
Explore systemwide cloud cost discovery