chore: import upstream snapshot with attribution

2026-07-13 13:10:45 +08:00
commit 4b6817381b
3933 changed files with 525247 additions and 0 deletions
@@ -0,0 +1,341 @@
+---
+title: "Debugging nf-core demo pipeline"
+description: "Using Tracer to diagnose and optimize UMI-based consensus sequencing"
+---
+
+This tutorial walks a bioinformatics engineer through real-time observability of the nf-core/fastquorum pipeline using Tracer's eBPF-powered monitoring. We simulate a small but realistic UMI-based duplex sequencing workflow on a single chromosome (chr17.fa), run it in a GitHub Codespace, and use Tracer to detect resource bottlenecks, identify redundant I/O, and explain why the pipeline completed in 1m 36s despite only 12 processes.
+
+## What You'll Learn
+
+- Connect a live Codespace to the Tracer sandbox
+- Auto-instrument a Nextflow pipeline with zero code changes
+- Visualize per-process CPU, memory, and I/O in real time
+- Extract actionable optimization insights
+
+<Info>
+    **Why this matters:** fastquorum is complex (UMI grouping, consensus
+    calling, dual alignment). Without OS-level visibility, engineers guess where
+    time is spent. Tracer shows exactly which process is the bottleneck — no
+    logs, no profiling flags.
+</Info>
+
+## Tools Used
+
+- **Pipeline:** nf-core/fastquorum v1.0.0+
+- **Environment:** GitHub Codespaces (Ubuntu 22.04, 4-core, 16GB RAM)
+- **Observability:** Tracer.bio (eBPF)
+- **Container:** Docker
+- **Genome:** chr17.fa (subset of GRCh38)
+
+## 1. Login & Setup: Tracer Sandbox + GitHub Codespaces
+
+We begin in a GitHub Codespace — a reproducible, cloud-based dev environment that mimics a local VM. Tracer's eBPF agent runs natively here and streams metrics to the Tracer Sandbox Dashboard (https://dev.sandbox.tracer.cloud) in real time.
+
+<Steps>
+  <Step title="Open GitHub Codespaces">
+    1. Go to [GitHub Codespaces](https://github.com/codespaces)
+    2. Click **"New codespace"**
+    3. Select **"Create your own"** → Paste this repo: `https://github.com/yourusername/nfcore-fastquorum-tracer-demo`
+    4. Choose machine: **4-core, 16GB RAM** (required for Docker + Nextflow)
+    5. Click **Create codespace**
+
+    ![Codespace display with the cloned nf-core pipeline repository](/images/tutorials/observability-driven/tracer-fig-1.webp)
+
+    *Fig 1: Codespace display with the cloned nf-core pipeline repository*
+
+  </Step>
+
+  <Step title="Install Tracer (One-Liner with Dev Branch & User Token)">
+    In the Codespaces terminal, run:
+
+    ```bash
+    curl -sSL https://install.tracer.cloud | CLI_BRANCH=dev sh -s user_35Fukh3QxSAxJLgfyE9SwPoPy9K
+    ```
+
+  </Step>
+
+  <Step title="Start Tracer Agent">
+    To start tracking a pipeline, run the following command:
+
+    ```bash
+    tracer init --token eyJh---- (your token)
+    ```
+
+    ![Successful connection snapshot 1](/images/tutorials/observability-driven/successful-connection-1.webp)
+
+    ![Successful connection snapshot 2](/images/tutorials/observability-driven/successful-connection-2.webp)
+
+    ![Successful connection snapshot 3](/images/tutorials/observability-driven/successful-connection-3.webp)
+
+    *Fig 2: You will see something like this upon successful connection (Snapshot of tracer init command which connecting to tracer)*
+
+    <Tip>
+    With the Tracer agent connected, input validated, and genome indexed, we now execute the full nf-core/fastquorum pipeline. No code changes are required — Tracer's eBPF hooks automatically detect nextflow launches, label processes, and stream OS-level metrics (CPU, RAM, I/O, syscalls) to your sandbox dashboard in real time.
+    </Tip>
+
+  </Step>
+</Steps>
+
+## 2. Dataset Preparation
+
+This section is critical — nf-core/fastquorum enforces strict requirements on input format, UMI placement, and file integrity.
+
+### Key Preparation Steps
+
+<Accordion title="Download Test Data">
+    We begin by downloading real test data directly from the nf-core
+    test-datasets repository, ensuring authenticity and compatibility.
+</Accordion>
+
+<Accordion title="Inspect FASTQ Files">
+    Confirm UMI structure — in this case, a 6-base inline UMI (NNNNNN) embedded
+    at the start of Read 1, which matches the expected pattern for duplex
+    consensus sequencing.
+</Accordion>
+
+<Accordion title="Validate File Paths">
+    Ensure all FASTQs are properly gzipped and accessible via relative paths to
+    avoid runtime errors.
+</Accordion>
+
+<Accordion title="Create Samplesheet">
+    A correctly formatted `samplesheet.csv` is constructed with mandatory
+    columns: `sample`, `fastq_1`, `fastq_2`, `umi_read`, and `umi_pattern`,
+    adhering to the pipeline's JSON schema.
+</Accordion>
+
+<Accordion title="Pre-build Genome Index">
+    To eliminate I/O noise during the observed run, the genome index (BWA-MEM1,
+    SAMtools FAIDX, and DICT) is pre-built locally and stored for reuse,
+    ensuring clean, reproducible eBPF telemetry from Tracer.
+</Accordion>
+
+## 3. Launch the Pipeline
+
+From the pipeline root:
+
+```bash
+nextflow run . \
+  --input samplesheet.csv \
+  --fasta data/chr17.fa \
+  --outdir results \
+  --duplex_seq true \
+  -profile test,docker \
+  -with-trace \
+  -with-report results/report.html
+```
+
+### Parameters
+
+| Flag                               | Purpose                          |
+| ---------------------------------- | -------------------------------- |
+| `--input samplesheet.csv`          | Validated manifest               |
+| `--fasta data/chr17.fa`            | Local reference                  |
+| `--duplex_seq true`                | Enable duplex consensus          |
+| `-profile test,docker`             | Use test config + containers     |
+| `-with-trace`                      | Nextflow-native trace (optional) |
+| `-with-report results/report.html` | HTML execution report            |
+
+## 4. Live Visualization: Tracer Dashboard During Execution
+
+With the nf-core/fastquorum pipeline launched and Tracer's eBPF agent actively streaming OS-level events, the Tracer Sandbox Dashboard becomes a real-time observability cockpit. No polling, no logs — just continuous, kernel-level telemetry delivered via WebSocket every 2 seconds.
+
+### Dashboard Entry Point: Run Overview
+
+Upon launching `nextflow run .`, a new run card appears instantly:
+
+**Run Overview Card:**
+
+- **Run Name:** run_1
+- **Status:** Running (blue dot)
+- **Elapsed:** 45s and counting
+- **Max RAM:** 12 / 100% → 12 GB peak (of 16 GB available)
+- **Avg. CPU:** 36 / 100% → 36% average across 4 cores
+- **Disk I/O:** 17 / 100% → 17% of max bandwidth
+
+![Run Overview Snapshot](/images/tutorials/observability-driven/run-overview.webp)
+
+_Fig 3: Run Overview Snapshot_
+
+![Compact Summary](/images/tutorials/observability-driven/compact-summary.webp)
+
+<Info>
+    This compact summary is the first signal that Tracer has auto-detected the
+    Nextflow executor and attached to all child processes — no `-with-trace` or
+    config changes needed. The progress bar fills as tasks complete, and
+    resource meters update in real time.
+</Info>
+
+### System Specs & Cost Panel
+
+| Metric         | Value                   | Status                 |
+| -------------- | ----------------------- | ---------------------- |
+| **RAM**        | 2.97 GB used / 15.62 GB | HEALTHY                |
+| **CPU**        | 1.81 cores / 4 cores    | HEALTHY                |
+| **DISK**       | 42.90 GB / 207.35 GB    | HEALTHY                |
+| **GPU**        | Not detected            | —                      |
+| **TOTAL COST** | $0.00                   | Free tier (Codespaces) |
+
+![System Specs & Cost Panel](/images/tutorials/observability-driven/specs&cost.webp)
+
+<Tip>
+    This panel confirms the GitHub Codespaces environment: a 4-core, 16 GB VM
+    with ample headroom. The cost meter at $0.00 reflects that this is a
+    non-billable sandbox run, but in production (e.g., AWS EC2), Tracer would
+    estimate hourly cost based on instance type and utilization.
+</Tip>
+
+### Tool Table: Real-Time Process Monitoring
+
+**Table Observations:**
+
+- `bwa index` is still running — expected: indexing chr17.fa (~80MB) is CPU-heavy
+- FastQC hit 118% CPU → Java thread burst (common in multi-threaded mode)
+- `samtools faidx` is I/O-light — just reads the FASTA once
+- Status badges update live: Running → Success as tasks finish
+
+**Visual Insights:**
+
+- **Critical path:** bwa index → FastqToBam → GroupReadsByUmi
+- **Parallelism:** samtools faidx and dict run concurrently with FastQC
+- **Tail latency:** Final MultiQC runs alone
+
+<Info>
+    This Gantt view is interactive — hover to see exact command, stdout, and
+    resource curve.
+</Info>
+
+| Tool             | Status  | Runtime  | Max RAM | Max CPU | Max Disk I/O |
+| ---------------- | ------- | -------- | ------- | ------- | ------------ |
+| bwa index        | Running | 9s 851ms | 0.12 GB | 115.49% | 0.04 GB      |
+| samtools faidx   | Success | 482ms    | 0.00 GB | 38.10%  | 0.00 GB      |
+| samtools dict    | Success | 1s 111ms | 0.08 GB | 54.63%  | 0.08 GB      |
+| FastQC           | Success | 5s 775ms | 0.30 GB | 118.23% | 0.01 GB      |
+| fgbio FastqToBam | Success | 4s 813ms | 0.14 GB | 120.60% | 0.00 GB      |
+
+![Timeline view](/images/tutorials/observability-driven/timeline-view.webp)
+
+![Table and visual insights for the tools running in pipeline at real-time](/images/tutorials/observability-driven/tool-table.webp)
+
+_Fig 4: Table (detailed) and visual insights for the tools running in pipeline at real-time_
+
+### Metrics Over Time: System-Level Trends
+
+**CPU Usage:**
+
+- Avg: 91.4%
+- Max: 115.5% (burst during bwa index)
+- Pattern: High at start (indexing), drops to ~70% during alignment
+
+**Memory Usage:**
+
+- Avg: 99.8 MB
+- Max: 121.5 MB
+- Spike at 6s: fgbio FastqToBam loads both FASTQs into memory
+
+**Disk I/O:**
+
+- Avg: 0.08 GB
+- Max: 0.18 GB
+- Burst at 40s: Writing intermediate BAM files
+
+**Network I/O:**
+
+- Avg: 81.42 MB
+- Max: 180.80 MB
+- Cause: Docker pulling nf-core/fastquorum:1.2.0 layers (first run)
+
+![CPU, Memory, Disk, Network Over Time](/images/tutorials/observability-driven/metrics-over-time.webp)
+
+![Metrics Over Time 2](/images/tutorials/observability-driven/metrics-over-time-2.webp)
+
+_Fig 5,6: System level trend_
+
+## 5. Post-Run Analysis: Resource Heatmap & Bottleneck Detection
+
+The pipeline completes in **1m 36s** with **12 successful tasks**. Now we analyze the full trace.
+
+### Resource Analysis
+
+| Process              | CPU (avg) | RAM (peak) | I/O (total) | Duration |
+| -------------------- | --------- | ---------- | ----------- | -------- |
+| BWAMEM1_INDEX        | 95%       | 1.4 GB     | 180 MB      | 53s      |
+| GROUPREADSBYUMI      | 99%       | 3.1 GB     | 42 MB       | 24s      |
+| CALLDDUPLEXCONSENSUS | 60%       | 1.8 GB     | 28 MB       | 16s      |
+| FASTQTOBAM           | 75%       | 1.2 GB     | 35 MB       | 18s      |
+
+### Key Insights
+
+<CardGroup cols={2}>
+  <Card title="Critical Path Identified" icon="route">
+    BWAMEM1_INDEX (53s) is the bottleneck — accounts for 55% of total runtime
+  </Card>
+  <Card title="Memory Spike" icon="memory">
+    GROUPREADSBYUMI peaks at 3.1 GB — consider increasing memory allocation for larger datasets
+  </Card>
+  <Card title="CPU Efficiency" icon="microchip">
+    Most processes utilize >75% CPU — good parallelization
+  </Card>
+  <Card title="I/O Optimization" icon="hard-drive">
+    Total I/O: 285 MB — minimal disk bottleneck detected
+  </Card>
+</CardGroup>
+
+## 6. Conclusion
+
+In the fast-evolving landscape of bioinformatics, where pipelines demand precision amid mounting computational complexity, **Tracer emerges as an indispensable ally** for bioinformaticians seeking deeper, actionable insights without the burden of invasive instrumentation.
+
+### Key Benefits
+
+By harnessing **eBPF technology** at the operating system level, Tracer delivers:
+
+- **Real-time observability** into every facet of your workflows (Nextflow, WDL, Bash, or CWL)
+- **Automatic detection** of hangs, crashes, and silent failures that traditional logs often overlook
+- **One-minute setup** with zero code modifications
+
+### Real-World Impact
+
+<Accordion title="Pinpoint Exact Failures">
+    Imagine pinpointing the exact genome file or tool process causing a crash in
+    a duplex sequencing run, or uncovering memory oversizing in dependency
+    updates that could shave weeks off troubleshooting.
+</Accordion>
+
+<Accordion title="Resource Optimization">
+    Tracer excels in resource orchestration, spotlighting inefficiencies like
+    redundant I/O in alignment steps or overprovisioned instances.
+</Accordion>
+
+<Accordion title="AI-Driven Recommendations">
+    AI-driven recommendations enable right-sizing of compute environments in
+    mere clicks, potentially slashing costs by 30% or more on cloud platforms,
+    paying only 5% of your pipeline's compute expenses without upfront fees.
+</Accordion>
+
+### Your Next Steps
+
+For bioinformaticians juggling high-throughput NGS data, evolving dependencies, and the pressure to derive reproducible insights from vast datasets, **Tracer isn't just a monitoring tool — it's a superpower** that shifts focus from infrastructure headaches to scientific discovery, fostering scalable, cost-effective workflows that accelerate breakthroughs in genomics, proteomics, and beyond.
+
+<Card title="Try Tracer Sandbox" icon="flask" href="https://tracer.bio">
+    Dive into the Tracer sandbox today and experience how effortless
+    observability can redefine your pipeline mastery.
+</Card>
+
+## Related Tutorials
+
+<CardGroup cols={2}>
+    <Card
+        title="Viewing Task Status"
+        href="/tutorials/viewing-task-status"
+        icon="eye"
+    >
+        Learn how to monitor task execution in real-time
+    </Card>
+    <Card
+        title="Investigating Task Failures"
+        href="/tutorials/investigating-task-failures"
+        icon="bug"
+    >
+        Debug and resolve failures with diagnostic tools
+    </Card>
+</CardGroup>
@@ -0,0 +1,96 @@
+---
+title: "Investigating Task Failures"
+description: "Debug and resolve failures with Tracer powerful diagnostic tools"
+---
+
+## Overview
+
+Tracer automatically captures logs, resource metrics, and system call data for every task, even those that fail.
+Therefore, when tasks fail, Tracer provides detailed information to help you understand what went wrong and how to fix it.
+
+Find out how to do this easily below.
+
+<video
+    autoPlay
+    loop
+    muted
+    playsInline
+    style={{ width: "100%", maxWidth: "100%" }}
+>
+    <source
+        src="/images/tutorials/failure-investigation/Investigating-failures.mp4"
+        type="video/mp4"
+    />
+    Your browser does not support the video tag.
+</video>
+
+## Previous Knowledge
+
+Before diving in how we can investigate task failures, it is recommended to have a basic understanding of the following concepts:
+
+<CardGroup cols={2}>
+    <Card
+        title="Navigate to Run Overview"
+        href="/tutorials/viewing-task-status"
+    >
+        See the run details
+    </Card>
+    <Card title="Viewing Task Status" href="/tutorials/viewing-task-status">
+        Monitoring your tasks
+    </Card>
+</CardGroup>
+
+## Identifying Failed Tasks
+
+<Steps>
+  <Step title="Run Overview">
+    When you see the run overview and notice a failed tool, you can click on the "Logs" tab to see the error summary and exit code.<br/>
+    ![Run Overview](/images/tutorials/failure-investigation/1.png)
+    <Info> As you can see in the image above, multiple tasks failed and therefore need to be investigated. </Info>
+  </Step>
+  <Step title="Choose the Log to Investigate">
+    Here, all logs of the failed run are shown. Select the log you want to investigate deeper.
+    ![Log Overview](/images/tutorials/failure-investigation/2.png)
+  </Step>
+  <Step title="Logs & Insights">
+    This page displays your log and its insights. It includes an error summary, plus automatic logs with warning and failure indicators.<br/>
+    Based on this data, our AI identifies likely root causes and tells you what happened, so you know exactly what needs attention. It also provides recommended solutions to resolve the issue.
+    ![Log Details](/images/tutorials/failure-investigation/3.png)
+  </Step>
+  <Step title="AI Log Analysis">
+    Our AI Log Analysis is divide up into three main sections:
+    1. **Critical Issue Section** - What exactly went wrong
+    2. **Next steps/Solution Suggestions** - How to resolve the issue and refrain from making the same mistake again
+    3. **Error Entries** - The specific lines in the log that caused the error
+
+![AI Insights](/images/tutorials/failure-investigation/4.png)
+
+  </Step>
+  <Step title="Log Details">
+    If you want to dig deeper into the logs, in this section you can see the full log with highlighted error lines.<br/>
+    On the right side, you can see the specific error entries that caused the task to fail together with the warning indicators. This also gives you the opportunity to download the full report.
+    ![Log Details](/images/tutorials/failure-investigation/5.png)
+    <Info> As logs can be very extensive, there are multiple ways of searching through the logs. You can use the search bar to search for specific keywords, or you can use the filter bar to filter for specific error types and you can filter on time as well. </Info>
+  </Step>
+</Steps>
+
+## Common Failure Patterns
+
+### Resource Exhaustion
+
+Tasks may fail due to insufficient resources:
+
+- **Out of Memory (OOM)** - Task exceeded available RAM
+- **Disk Space** - Insufficient storage for outputs
+- **CPU Timeout** - Task exceeded maximum execution time
+
+<Info>
+    Tracer's eBPF monitoring captures resource usage leading up to failures,
+    helping you identify resource constraints.
+</Info>
+
+## Next Steps
+
+<Card title="Viewing Task Status" href="/tutorials/viewing-task-status">
+    Learn how to monitor task execution
+</Card>
@@ -0,0 +1,129 @@
+---
+title: 'Viewing Task Status'
+description: 'Learn how to monitor and view the status of your tasks in Tracer'
+---
+
+On this page, we will explain how you can see the status of your tasks within a specific pipeline run.<br/>
+<Info>If you have not yet run a pipeline, please refer to the [quickstart](/quickstart) to get started.</Info>
+
+When executed correctly, you should be able to see the following screen in the Tracer dashboard:
+
+<video
+  autoPlay
+  loop
+  muted
+  playsInline
+  style={{ width: '100%', maxWidth: '100%' }}
+>
+  <source src="/images/tutorials/view-task-status/View-Tool-Details.mp4" type="video/mp4" />
+  Your browser does not support the video tag.
+</video>
+
+<Tip>Tracer updates task status in real time, so you can monitor progress without refreshing the page.</Tip>
+
+Tracer provides detailed visibility into task execution, allowing you to track progress, identify bottlenecks, and ensure your workflows are running smoothly.
+
+## Accessing Task Status
+
+<Steps>
+  <Step title="Navigate to Pipelines Overview">
+    Open the Tracer dashboard and select the **"Pipelines"** tab next to "Overview". This will show you all the pipelines that have been run.
+
+    ![General Overview](/images/tutorials/view-task-status/1.png)
+
+    <Tip>When hovering over the pipeline runs, you get more information about what pipelines have been run or are currently running.</Tip>
+  </Step>
+  <Step title="Go to your Specific Pipeline">
+    Access your specific pipeline by clicking on the pipeline name. You can recognize the pipelines which are currently running by the blue dot next to the pipeline name.
+
+    ![Pipeline Overview](/images/tutorials/view-task-status/2.png)
+  </Step>
+  <Step title="Select the Correct Run">
+    As you will most likely check the current or latest run of the selected pipeline, you can click the blue link **"Latest Run"** to go to the latest run.
+
+    If you are searching for an older run, you can click on the **"Pipeline Runs"** box in the bottom left corner of the screen. This will show you all runs, where you can select the run you are looking for.
+
+    ![Pipeline Runs](/images/tutorials/view-task-status/3.png)
+  </Step>
+  <Step title="Navigate to Tools">
+    Here, you get the first high-level overview of the pipeline run. You can see the main metrics, runtime and cost as well as the automatic logs with warnings and failure indicators.
+
+    To go to the task status, click on the **"Tools"** tab next to "Overview". This will show you the status of all tasks within this pipeline run.
+
+    ![Runs Overview](/images/tutorials/view-task-status/4.png)
+  </Step>
+  <Step title="Investigate Task Status">
+    The tool visualizer shows you the status of all tasks within this pipeline run:
+    - **Bar width** indicates the runtime of the task
+    - **Bar color** indicates the status: Grey (running), Green (completed), Red (failed)
+
+    ![Tool Visualiser](/images/tutorials/view-task-status/5.png)
+
+    <Info>In the image above, all tools are completed as indicated by the green bars.</Info>
+
+    **When hovering over a specific task**, you can see the status, runtime, and the exact moment the task started. By clicking the bar, you can dig one level deeper to see the metrics, runtime, cost, and logs with warnings and failure indicators.
+
+    ![Running Hover](/images/tutorials/view-task-status/6.png)
+
+    <Info>In the image above, the task is still running as indicated by the grey bar and has been running for exactly 30.591s.</Info>
+
+    **For a more detailed overview**, click on the **"Detailed"** button in the top right corner of the screen. This will list all tools in a table showing status, command, start time, runtime, max RAM, max CPU, max Disk I/O, and end time.
+
+    ![Detailed View](/images/tutorials/view-task-status/7.png)
+
+    <Info>In the image above, you can see that one task is still running as indicated by the "Running" status, while the others are successfully completed.</Info>
+  </Step>
+  <Step title="Tool Specific Metrics">
+    Here, you can see the run-specific metrics including:
+    - **Tool ID** - Unique identifier for the task
+    - **Run Name** - Name of the current run
+    - **Pipeline Name** - Name of the parent pipeline
+    - **AI Insights** - Intelligent analysis and recommendations
+    - **Runtime Metrics** - CPU, memory, and I/O usage
+
+    You can also compare this task's performance to previous runs.
+
+    ![Tool Metrics](/images/tutorials/view-task-status/8.png)
+  </Step>
+</Steps>
+
+**After these steps, you’ll have live visibility into task progress, metrics, and logs for your pipeline run.**
+
+
+## Task Status Indicators
+
+Tracer displays tasks with the following status indicators:
+
+| Status | Description | Color Used to Indicate |
+|--------|-------------| ----------------------|
+| **Running** | Task is currently executing | Grey |
+| **Completed** | Task finished successfully | Green |
+| **Failed** | Task encountered an error | Red |
+
+## Task Metrics
+
+When viewing a task, you'll see detailed information including:
+
+| Category | Metric | Description |
+|----------|--------|-------------|
+| **Status & Timing** | Status | Current state of the task (Running, Completed, Failed) |
+| | Runtime | Total execution time |
+| | Start Time | When the task began |
+| | End Time | When the task completed (if applicable) |
+| **Resource Usage** | CPU | Processor utilization |
+| | RAM Memory | Memory consumption |
+| | Disk I/O | Read/write operations |
+| | Network I/O | Network traffic |
+| **Logs & Debugging** | Automatic Logs | Standard output and deep system-level logs captured |
+| | Warnings | Highlighted warnings and failure indicators |
+| **Additional Information** | Command | The exact command executed |
+| | Historical Comparison | Performance comparison with previous runs of this task |
+
+
+## Next Steps
+
+How to get even more out of Tracer's capabilities:
+
+<Card title="Investigating Task Failures" href="/tutorials/investigating-task-failures">
+  Learn how to debug failed tasks
+</Card>