wehub/tensorflow--tensorflow
1
0
Fork 0
Code Issues Pull Requests Actions 5 Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution
cffconvert / validate (push) Has been skipped
Details
License Check / license-check (push) Failing after 2s
Details

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 12:14:16 +08:00
commit 8a852e4b4e
36502 changed files with 9277225 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
third_party/xla/docs/_toc.yaml
Vendored
+148
View File
@@ -0,0 +1,148 @@
# Copyright 2026 The OpenXLA Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================
toc:
- heading: XLA documentation
- title: Getting started
# This is the default tab for the Getting started section.
section:
- title: Introduction
path: /xla
- title: XLA architecture overview
path: /xla/architecture
- title: XLA terminology
path: /xla/terminology
- title: XLA flags guidance
path: /xla/flags_guidance
- title: Topical guides
# This is the default tab for the Topical guides section.
section:
- title: Journey from HLO to executable
path: /xla/hlo_to_thunks
# XLA Ops sub-section
- heading: XLA Ops
- title: Operation semantics
path: /xla/operation_semantics
- title: HLO Passes
path: /xla/hlo_passes
- title: Aliasing
path: /xla/aliasing
- title: Async HLO instructions
path: /xla/async_ops
# Data representation sub-section
- heading: Data representation
- title: Shapes and layout
path: /xla/shapes
- title: Tiled layout
path: /xla/tiled_layout
- title: Symbolic expressions and maps
path: /xla/symbolic_expression
- title: Indexing Analysis
path: /xla/indexing
- title: Broadcasting
path: /xla/broadcasting
# GPU backend sub-section
- heading: GPU backend
- title: GPU architecture
path: /xla/gpu_architecture
- title: GPU Emitters
path: /xla/emitters
- title: Persisted autotuning
path: /xla/persisted_autotuning
- title: Determinism (GPU)
path: /xla/determinism
# TPU backend sub-section
- heading: TPU backend
- title: Sparsecore (embeddings)
path: /xla/sparsecore
- title: Megascale
path: /xla/megascale/overview
# Performance sub-section
- heading: Performance
- title: LHS Cost Model
path: /xla/lhs_cost_model
- title: Effort Levels
path: /xla/effort_levels
# Advanced guides sub-section
- heading: Advanced guides
- title: Write custom calls
path: /xla/custom_call
- title: Develop a new backend for XLA
path: /xla/developing_new_backend
- title: Debugging
section:
# This is the default tab for the Debugging section.
- heading: XLA tooling
- title: Dump HLO computations
path: /xla/hlo_dumps
- title: Errors overview
path: /xla/errors_overview
- title: Error codes
path: /xla/error_codes
- title: Debug OOM errors with XProf
path: /xla/oom_debugging
- title: Debug Megascale
path: /xla/megascale/debugging_workflow
# XLA Tools sub-section
- heading: XLA tools
- title: XLA tooling
path: /xla/tools
- title: Multi-host HLO runner
path: /xla/tools_multihost_hlo_runner
- title: Contributing
# This is the default tab for the Contributing section.
section:
- title: Developer guide
path: /xla/developer_guide
- title: Set up development environment
path: /xla/developer_environment
- title: Build from source
path: /xla/build_from_source
- title: Testing HLO passes
path: /xla/test_hlo_passes
- title: Copybara quirks
path: /xla/copybara
- title: PJRT plugins
# This is the default tab for the PJRT section.
section:
- title: "PJRT - Uniform Device API"
path: /xla/pjrt/overview
- title: PJRT C++ API overview
path: /xla/pjrt/cpp_api_overview
- title: Develop a new PJRT plugin
path: /xla/pjrt/pjrt_integration
- title: PJRT examples
path: /xla/pjrt/examples
- title: XLA frontends
# This is the default tab for the XLA frontends section.
section:
- title: JAX
path: https://docs.jax.dev/en/latest/notebooks/thinking_in_jax.html
status: external
- title: "PyTorch/XLA"
path: https://docs.pytorch.org/xla/release/r2.8/index.html#pytorch-xla-documentation
status: external
- heading: TensorFlow
- title: Use XLA in TensorFlow
path: /xla/tf2xla
- title: Autoclustering tutorial
path: /xla/tf2xla/tutorials/autoclustering_xla
- title: Use XLA with tf.function
path: /xla/tf2xla/tutorials/jit_compile
third_party/xla/docs/aliasing.md
Vendored
+64
View File
@@ -0,0 +1,64 @@
# Aliasing in XLA
This document describes the XLA aliasing API, which lets you specify the
aliasing between the input and output buffers when building an XLA program.
## Defining aliasing at compile-time
For example, consider a trivial HLO module which simply adds `1` to its input:
```mlir
HloModule increment
ENTRY entry {
%p = f32[] parameter(0)
%c = f32[] constant(1)
ROOT %out = f32[] add(%p, %c)
}
```
This module will allocate two 4-byte buffers: one for the input `%p`, and one
for the output `%out`.
However, it is often desirable to perform the update in-place (for example, if
in the frontend generating the expression the input variable is no longer alive
after the computation, as in the increment `p++`).
To perform such an update efficiently, you can specify the input aliasing:
```mlir
HloModule increment, input_output_alias={ {}: 0 }
ENTRY entry {
%p = f32[] parameter(0)
%c = f32[] constant(1)
ROOT %out = f32[] add(%p, %c)
}
```
The format specifies that the entire output (marked by `{}`) is aliased to the
input parameter `0`.
To specify the aliasing programmatically, see the
[`XlaBuilder::SetUpAlias`](https://github.com/openxla/xla/blob/main/xla/client/xla_builder.h)
API.
## Defining aliasing at runtime
The aliasing defined in the previous step is specified during *compilation*.
During execution, you can use the
[`LocalClient::RunAsync`](https://github.com/openxla/xla/blob/main/xla/client/local_client.h)
API to choose whether to donate the buffer.
Input buffers to the program are wrapped in
[`ExecutionInput`](https://github.com/openxla/xla/blob/main/xla/service/executable.h)s,
which in turn contain a tree of `MaybeOwningDeviceMemory`. If memory is
specified as *owning* (ownership of the buffer is passed to the XLA runtime),
the buffer is actually donated, and the update is executed in place, as
requested by the compile-time aliasing API.
If, however, the buffer that is aliased at compile time is *not* donated at
runtime, *copy-protection* kicks in: an extra output buffer `O` is allocated,
and the contents of the input buffer `P` that was meant to be aliased are copied
into `O` (so effectively the program can execute as if the buffer `O` was
donated at runtime).
third_party/xla/docs/architecture.md
Vendored
+71
View File
@@ -0,0 +1,71 @@
# XLA architecture
XLA (Accelerated Linear Algebra) is a machine learning (ML) compiler that
optimizes linear algebra, providing improvements in execution speed and memory
usage. This page provides a brief overview of the objectives and architecture of
the XLA compiler.
## Objectives
Today, XLA supports several ML framework frontends (including PyTorch,
TensorFlow, and JAX) and is part of the OpenXLA project – an ecosystem of
open-source compiler technologies for ML that's developed collaboratively by
leading ML hardware and software organizations. Before the OpenXLA project was
created, XLA was developed inside the TensorFlow project, but the fundamental
objectives remain the same:
* **Improve execution speed.** Compile subgraphs to reduce the execution time
of short-lived ops and eliminate overhead from the runtime, fuse pipelined
operations to reduce memory overhead, and specialize known tensor shapes to
allow for more aggressive constant propagation.
* **Improve memory usage.** Analyze and schedule memory usage, eliminating
many intermediate storage buffers.
* **Reduce reliance on custom ops.** Remove the need for many custom ops by
improving the performance of automatically fused low-level ops to match the
performance of custom ops that were originally fused by hand.
* **Improve portability.** Make it relatively easy to write a new backend for
novel hardware, so that a large fraction of ML models can run unmodified on
that hardware. This is in contrast with the approach of specializing
individual monolithic ops for new hardware, which requires models to be
rewritten to make use of those ops.
## How it works
The XLA compiler takes model graphs from ML frameworks defined in
[StableHLO](https://github.com/openxla/stablehlo) and compiles them into machine
instructions for various architectures. StableHLO defines a versioned operation
set (HLO = high level operations) that provides a portability layer between ML
frameworks and the compiler.
In general, the compilation process that converts the model graph into a
target-optimized executable includes these steps:
1. XLA performs several built-in optimization and analysis passes on the
StableHLO graph that are target-independent, such as
[CSE](https://en.wikipedia.org/wiki/Common_subexpression_elimination),
target-independent operation fusion, and buffer analysis for allocating
runtime memory for the computation. During this optimization stage, XLA also
converts the StableHLO dialect into an internal HLO dialect.
2. XLA sends the HLO computation to a backend for further HLO-level
optimizations, this time with target-specific information and needs in mind.
For example, the GPU backend may perform operation fusions that are
beneficial specifically for the GPU programming model and determine how to
partition the computation into streams. At this stage, backends may also
pattern-match certain operations or combinations thereof to optimized
library calls.
3. The backend then performs target-specific code generation. The CPU and GPU
backends included with XLA use [LLVM](http://llvm.org) for low-level IR,
optimization, and code generation. These backends emit the LLVM IR necessary
to represent the HLO computation in an efficient manner, and then invoke
LLVM to emit native code from this LLVM IR.
Within this process, the XLA compiler is modular in the sense that it is easy to
slot in an alternative backend to
[target some novel HW architecture](./developing_new_backend.md). The GPU
backend currently supports NVIDIA GPUs via the LLVM NVPTX backend. The CPU
backend supports multiple CPU ISAs.
third_party/xla/docs/async_ops.md
Vendored
+256
View File
@@ -0,0 +1,256 @@
# Async HLO Instructions
1. Adding async operations to HLO is cumbersome (i.e. `all-reduce-start` and
`all-reduce-done`).
2. The start and done split may be inadequate for some of the asynchronous use
cases.
To target the first shortcoming, we propose to introduce one last set of new
asynchronous opcodes: `kAsyncStart`, `kAsyncUpdate`, and `kAsyncDone`. The idea
is to create a generic asynchronous opcode that can wrap any HLO instruction.
The actual operation that will be performed asynchronously will be encoded using
a called computation that only has the instruction as its root and any
parameters for inputs. The in-flight input/output buffer handling and aliasing
can then be shared for any asynchronous operation. The async-start instruction’s
output shape will then be a tuple of the input operands, output values, and any
intermediate state that is needed for the `async-update` or `async-done`
instructions.
```
%async_op {
%param0 = f32[64] parameter(0)
ROOT %op = f32[32] op(f32[64] %param0), op_specific_attr=”foo”
}
%async-start = ((f32[64]), f32[32], s32[]) async-start(f32[64] %operand),
calls=%async_op
%async-done = f32[32] async-done(((f32[64]), f32[32], s32[]) %async-start)
```
In the representation above, only `async-start` has a called computation since
it is trivial to find what the `async-done` does by following its operand to
find the corresponding `async-start` to find the called computation.
Also note that the first element in the output tuple of `async-start` is a
tuple containing the operands. The elements of this operand tuple alias with
the respective operands, so their buffers stay alive until at least the
`async-done` instruction. Similarly, the second element aliases with the output
of `async-done`, and the third element is the context state that is used to
keep track of the asynchronous operation. This representation naturally
supports multiple tensors in the asynchronous operation input and/or output:
```
%async_op {
%param0 = f32[64] parameter(0)
%param1 = f32[64] parameter(1)
ROOT %op = (f32[32], f32[32]) op(f32[64] %param0, f32[64] %param1),
op_specific_attr=”foo”
}
%async-start = ((f32[64], f32[64]), (f32[32], f32[32]), s32[])
async-start(f32[64] %operand0, f32[64] %operand1),
calls=%async_op
%async-done = (f32[32], f32[32]) async-done(%async-start)
```
In addition, the op can further be decomposed into zero or more `async-update`
steps that perform intermediate computations. The input/output aliasing works
the same way with the `async-update` instruction and each `async-start` and
`async-update` instructions must have one user that is either another
`async-update` or an `async-done`:
```
%async_op {
%param0 = f32[64] parameter(0)
ROOT %op = f32[32] op(f32[64] %param0), op_specific_attr=”foo”
}
%async-start = ((f32[64]), f32[32], s32[]) async-start(f32[64] %operand),
calls=%async_op
%async-update0 = ((f32[64]), f32[32], s32[]) async-update(
((f32[64]), f32[32], s32[]) %async-start)
%async-update1 = ((f32[64]), f32[32], s32[]) async-update(
((f32[64]), f32[32], s32[]) %async-update0)
%async-done = f32[32] async-done(((f32[64]), f32[32], s32[]) %async-update1)
```
## Syntax sugar
The HLO parser supports syntax sugar to automatically parse and print
asynchronous operations as if they are first-class opcodes. The parser treats
the `-start`, `-update`, and `-done` suffixes specially by automatically
creating the async computation and the wrapped instruction (without the suffix).
For example, an asynchronous `custom-call` can be written as:
```
%cc-start = ((f32[64]), f32[32], s32[]) custom-call-start(%operand),
custom_call_target="foo"
%cc-update = ((f32[64]), f32[32], s32[]) custom-call-update(%cc-start)
%result = f32[32] custom-call-done(%cc-update)
```
The parser desugars this into the following equivalent HLO:
```
%async_computation {
%p0 = f32[64] parameter(0)
ROOT %custom-call = f32[32] custom-call(%p0), custom_call_target="foo"
}
%async-start = ((f32[64]), f32[32], s32[]) async-start(%operand),
calls=%async_computation
%async-update = ((f32[64]), f32[32], s32[]) async-update(%async-start)
%result = f32[32] async-done(%async-update)
```
This desugaring is supported for most HLO opcodes (e.g., `custom-call`, `dot`,
`all-reduce`, etc.).
### Exceptions
In order not to create ambiguities, the parser will not desugar operations that
have explicit first-class opcodes defined with the `-start` and/or `-done`
suffixes (e.g., `copy-start`/`copy-done`,
`collective-permute-start`/`collective-permute-done`). These will continue to
use their respective first-class opcodes.
## Late Binding
In some cases, the operands (inputs) or outputs of an asynchronous
operation are not all available or allocated when the operation starts.
XLA supports *late binding*, which allows operands to be incrementally
bound during `async-update` steps, and outputs to be bound during either
`async-update` or `async-done` steps.
### Representation in HLO
For a called computation that expects $N$ parameters, we can start the
asynchronous execution with fewer than $N$ operands. The remaining
operands are passed in subsequent `async-update` instructions.
* `async-start` binds the first $K$ operands ($K < N$).
* `async-update` instructions bind the remaining $N - K$ operands.
Operand bindings must happen in left-to-right order. That is, if a
computation expects parameters $P_0, P_1, \dots, P_{N-1}$, they must be bound in
that order across the async chain.
The `async-start` and `async-update` shapes reflect the incrementally
bound parameters. Specifically, the first element of the tuple shape
(the operand shapes) grows as more operands are bound.
Output binding is independent of operand binding and can happen at any
step in the async chain (either in an `async-update` or at the final
`async-done`).
### Example with `kCall`
Consider a called computation `%foo` that takes two parameters:
```
%foo {
%p0 = f32[] parameter(0)
%p1 = f32[] parameter(1)
ROOT %add = f32[] add(%p0, %p1)
}
```
We can call this computation asynchronously, binding `%p0` at start and
`%p1` at update:
```
%call-start = ((f32[]), (), s32[]) call-start(%operand0), to_apply=%foo
%call-update = ((f32[], f32[]), f32[], s32[]) call-update(%call-start, %operand1)
%result = f32[] call-done(%call-update)
```
The parser desugars this into the following HLO:
```
%async-start = ((f32[]), (), s32[]) async-start(%operand0), calls=%foo
%async-update = ((f32[], f32[]), f32[], s32[]) async-update(%async-start, %operand1)
%result = f32[] async-done(%async-update)
```
### Late-Bound Outputs
In addition to operands (inputs), the **outputs** of an asynchronous
operation can also be bound late. This is useful when the output
buffers are not known or allocated at the start of the operation.
To represent late-bound outputs:
1. The `async-start` (or `call-start`) instruction is defined with an
empty tuple `()` at index 1 of its output shape (the result slot).
2. A subsequent `async-update` (or `call-update`) instruction
specifies the actual output shape at index 1, replacing the empty
tuple.
3. Alternatively, the output can be bound at the end of the chain by
the `async-done` (or `call-done`) instruction, which returns the
final output shape. This can be done regardless of whether there are
intermediate `async-update` steps in the chain.
#### Example with `async-update`
```
// Output is not bound at start (index 1 is ())
%call-start = ((f32[1024]), (), s32[]) call-start(%input_buffer), to_apply=%foo
// Output is bound at update (index 1 becomes (f32[1024]))
%call-update = ((f32[1024]), (f32[1024]), s32[]) call-update(%call-start, %output_buffer)
%result = (f32[1024]) call-done(%call-update)
```
The parser desugars this into:
```
%async-start = ((f32[1024]), (), s32[]) async-start(%input_buffer), calls=%foo
%async-update = ((f32[1024]), (f32[1024]), s32[]) async-update(%async-start, %output_buffer)
%result = (f32[1024]) async-done(%async-update)
```
#### Example with `async-done` (without `async-update`)
If there are no intermediate update steps, the output can be bound
directly at `async-done`:
```
// Output is not bound at start (index 1 is ())
%call-start = ((f32[1024]), (), s32[]) call-start(%input_buffer), to_apply=%foo
// Output is bound at done
%result = (f32[1024]) call-done(%call-start)
```
The parser desugars this into:
```
%async-start = ((f32[1024]), (), s32[]) async-start(%input_buffer), calls=%foo
%result = (f32[1024]) async-done(%async-start)
```
#### Example with intermediate `async-update` and output bound at `async-done`
If there are intermediate update steps to bind operands, but the output
is still bound at the very end:
```
// Output is not bound at start, no operands bound
%call-start = ((), (), s32[]) call-start(), to_apply=%foo
// Operands are bound at update, but output remains unbound (index 1 is ())
%call-update = ((f32[], f32[]), (), s32[]) call-update(%call-start, %operand0, %operand1)
// Output is bound at done
%result = f32[] call-done(%call-update)
```
The parser desugars this into:
```
%async-start = ((), (), s32[]) async-start(), calls=%foo
%async-update = ((f32[], f32[]), (), s32[]) async-update(%async-start, %operand0, %operand1)
%result = f32[] async-done(%async-update)
```
third_party/xla/docs/broadcasting.md
Vendored
+206
View File
@@ -0,0 +1,206 @@
# Broadcasting
This document describes the broadcasting semantics of XLA.
## What is broadcasting?
Broadcasting is the process of making arrays with different shapes have
compatible shapes for arithmetic operations. The terminology is borrowed from
[NumPy broadcasting](http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html).
Broadcasting may be required for operations between multi-dimensional arrays of
different ranks, or between multi-dimensional arrays with different but
compatible shapes. Consider the addition `X+v` where `X` is a matrix (an array
with 2 dimensions) and `v` is a vector (an array with 1 dimension). To perform
element-wise addition, XLA needs to "broadcast" the vector `v` to the same
number of dimensions as the matrix `X`, by replicating `v` a certain number of
times. The vector's length has to match at least one of the dimensions of the
matrix.
For example:
|1 2 3| + |7 8 9|
|4 5 6|
The matrix's dimensions are (2,3), and the vector's dimension is (3). The vector
is broadcast by replicating it over rows to get:
|1 2 3| + |7 8 9| = |8 10 12|
|4 5 6| |7 8 9| |11 13 15|
In NumPy, this is called
[broadcasting](http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html).
## Principles
The XLA language is as strict and explicit as possible, avoiding implicit
"magical" features. Such features might make some computations slightly easier
to define, but at the cost of more assumptions baked into user code that will be
difficult to change in the long term. If necessary, implicit magical features
can be added in client-level wrappers.
With regard to broadcasting, XLA requires explicit broadcasting specifications
on operations between arrays of different ranks. This is different from NumPy,
which infers the specification when possible.
## Broadcasting a lower-dimensional array onto a higher-dimensional array
*Scalars* can always be broadcast over arrays without an explicit specification
of broadcasting dimensions. An element-wise binary operation between a scalar
and an array means applying the operation with the scalar to each element in the
array. For example, adding a scalar to a matrix means producing a matrix in
which each element is a sum of the scalar and the corresponding element of the
input matrix.
|1 2 3| + 7 = |8 9 10|
|4 5 6| |11 12 13|
Most broadcasting needs can be captured by using a tuple of dimensions on a
binary operation. When the inputs to the operation have different ranks, this
broadcasting tuple specifies which dimension(s) in the **higher-dimensional**
array to match with the **lower-dimensional** array.
Consider the previous example. Instead of adding a scalar to a (2,3) matrix, add
a vector of dimension (3) to a matrix of dimensions (2,3). *Without specifying
broadcasting, this operation is invalid.* To correctly request matrix-vector
addition, specify the broadcasting dimension to be (1), meaning the vector's
dimension is matched to dimension 1 of the matrix. In 2D, if dimension 0
represents rows and dimension 1 represents columns, this means that each element
of the vector becomes a column of a size matching the number of rows in the
matrix:
|7 8 9| ==> |7 8 9|
|7 8 9|
As a more complex example, consider adding a 3-element vector (dimension (3)) to
a 3x3 matrix (dimensions (3,3)). There are two ways broadcasting can happen for
this example:
(1) A broadcasting dimension of 1 can be used. Each vector element becomes a
column and the vector is duplicated for each row in the matrix.
|7 8 9| ==> |7 8 9|
|7 8 9|
|7 8 9|
(2) A broadcasting dimension of 0 can be used. Each vector element becomes a row
and the vector is duplicated for each column in the matrix.
|7| ==> |7 7 7|
|8| |8 8 8|
|9| |9 9 9|
Note: When adding a 2x3 matrix to a 3-element vector, a broadcasting dimension
of 0 is invalid.
The broadcasting dimensions can be a tuple that describes how a
smaller-dimensional shape is broadcast into a larger-dimensional shape. For
example, given a 2x3x4 cuboid and a 3x4 matrix, a broadcasting tuple (1,2)
means matching the matrix to dimensions 1 and 2 of the cuboid.
This type of broadcast is used in the binary ops in `XlaBuilder`, if the
`broadcast_dimensions` argument is given. For example, see
[XlaBuilder::Add](https://github.com/openxla/xla/blob/main/xla/hlo/builder/xla_builder.cc).
In the XLA source code, this type of broadcasting is sometimes called "InDim"
broadcasting.
### Formal definition
The broadcasting attribute allows matching a lower-dimensional array to a
higher-dimensional array by specifying which dimensions of the
higher-dimensional array to match. For example, for an array with dimensions
MxNxPxQ, a vector with dimension T can be matched as follows:
MxNxPxQ
dim 3: T
dim 2: T
dim 1: T
dim 0: T
In each case, T has to be equal to the matching dimension of the
higher-dimensional array. The vector's values are then broadcast from the
matched dimension to all the other dimensions.
To match a TxV matrix onto the MxNxPxQ array, a pair of broadcasting dimensions
is used:
MxNxPxQ
dim 2,3: T V
dim 1,2: T V
dim 0,3: T V
etc...
The order of dimensions in the broadcasting tuple must be the order in which the
dimensions of the lower-dimensional array are expected to match the dimensions
of the higher-dimensional array. The first element in the tuple specifies which
dimension in the higher-dimensional array has to match dimension 0 in the
lower-dimensional array. The second element in the tuple specifies which
dimension in the higher-dimensional array has to match dimension 1 in the
lower-dimensional array, and so on. The order of broadcast dimensions must be
strictly increasing. For example, in the previous example it is illegal to
match V to N and T to P; it is also illegal to match V to both P and N.
## Broadcasting similar-dimensional arrays with degenerate dimensions
A related problem is broadcasting two arrays that have the same number of
dimensions but different dimension sizes. As with NumPy, this is only possible
when the arrays are *compatible*. Two arrays are compatible when all their
dimensions are compatible. Two dimensions are compatible if:
* They are equal, or
* One of them is 1 (a "degenerate" dimension)
When two compatible arrays are encountered, the result shape has the maximum of
the two inputs at every dimension index.
Examples:
1. (2,1) and (2,3) broadcast to (2,3).
2. (1,2,5) and (7,2,5) broadcast to (7,2,5).
3. (7,2,5) and (7,1,5) broadcast to (7,2,5).
4. (7,2,5) and (7,2,6) are incompatible and cannot be broadcast.
A special case arises, and is also supported, where each of the input arrays has
a degenerate dimension at a different index. In this case, the result is an
"outer operation": (2,1) and (1,3) broadcast to (2,3). For more examples,
consult the
[NumPy documentation on broadcasting](http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html).
## Broadcast composition
Broadcasting of a lower-dimensional array to a higher-dimensional array **and**
broadcasting using degenerate dimensions can both be performed in the same
binary operation. For example, a vector of size 4 and a matrix of size 1x2 can
be added together using broadcast dimensions of value (0):
|1 2 3 4| + [5 6] // [5 6] is a 1x2 matrix, not a vector.
First the vector is broadcast up to 2-dimensional (matrix) using the broadcast
dimensions. The single value (0) in the broadcast dimensions indicates that
dimension zero of the vector matches dimension zero of the matrix. This produces
a matrix of size 4xM where the value M is chosen to match the corresponding
dimension size in the 1x2 array. Therefore, a 4x2 matrix is produced:
|1 1| + [5 6]
|2 2|
|3 3|
|4 4|
Then "degenerate dimension broadcasting" broadcasts dimension zero of the 1x2
matrix to match the corresponding dimension size of the right hand side:
|1 1| + |5 6| |6 7|
|2 2| + |5 6| = |7 8|
|3 3| + |5 6| |8 9|
|4 4| + |5 6| |9 10|
A more complicated example is a matrix of size 1x2 added to an array of size
4x3x1 using broadcast dimensions of (1, 2). First the 1x2 matrix is broadcast up
to 3 dimensions using the broadcast dimensions to produce an intermediate Mx1x2
array where the dimension size M is determined by the size of the larger operand
(the 4x3x1 array) producing a 4x1x2 intermediate array. The M is at dimension 0
(the left-most dimension) because the dimensions 1 and 2 are mapped to the
dimensions of the original 1x2 matrix as the broadcast dimensions are (1, 2).
This intermediate array can be added to the 4x3x1 matrix using broadcasting of
degenerate dimensions to produce a 4x3x2 array result.
third_party/xla/docs/build_from_source.md
Vendored
+231
View File
@@ -0,0 +1,231 @@
# Build from source
This document describes how to build XLA components.
If you did not clone the XLA repository or install Bazel, check out the initial
sections of the [XLA Developer Guide](developer_guide.md).
## Linux
### Configure
XLA builds are configured by the `.bazelrc` file in the repository's root
directory. The `./configure.py` script can be used to adjust common settings.
If you need to change the configuration, run the `./configure.py` script from
the repository's root directory. This script has flags for the location of XLA
dependencies and additional build configuration options (compiler flags, for
example). Refer to the *Sample session* section for details.
### CPU support
We recommend using a suitable Docker image - such as
[ml-build](https://us-docker.pkg.dev/ml-oss-artifacts-published/ml-public-container/ml-build)
, which is also used in XLA's CI workflows on GitHub - for building and testing
XLA. The ml-build image comes with Clang 18 pre-installed.
```sh
docker run -itd --rm \
--name xla \
-w /xla \
-v $PWD:/xla \
us-docker.pkg.dev/ml-oss-artifacts-published/ml-public-container/ml-build:latest \
bash
```
Using a Docker container, you can build XLA with CPU support by running the
following commands:
```sh
docker exec xla ./configure.py --backend=CPU
docker exec xla bazel build \
--spawn_strategy=sandboxed \
--test_output=all \
//xla/...
```
If you want to build XLA targets with CPU support **without using Docker**,
you’ll need to install Clang. XLA is currently built with Clang 18 in CI,
but earlier versions should also work.
To configure and build the targets, run the following commands:
```sh
./configure.py --backend=CPU
bazel build \
--spawn_strategy=sandboxed \
--test_output=all \
//xla/...
```
### GPU support
We recommend using the same Docker container mentioned above to build XLA with
GPU support.
To start Docker container with access to all GPUs, run the following command:
```sh
docker run -itd --rm \
--gpus all \
--name xla_gpu \
-w /xla \
-v $PWD:/xla \
us-docker.pkg.dev/ml-oss-artifacts-published/ml-public-container/ml-build:latest \
bash
```
To build XLA with GPU support, run the following commands:
```sh
docker exec xla_gpu ./configure.py --backend=CUDA
docker exec xla_gpu bazel build \
--spawn_strategy=sandboxed \
--test_output=all \
//xla/...
```
**Note:** You can build XLA on a machine without GPUs. In that case:
- Do **not** use `--gpus all` flag when starting the Docker container.
- Specify CUDA compute capabilities manually, For example:
```
docker exec xla_gpu ./configure.py --backend=CUDA \
--cuda_compute_capabilities="9.0"
```
For more details regarding
[TensorFlow's GPU docker images you can check out this document.](https://www.tensorflow.org/install/source#gpu_support_2)
You can build XLA targets with GPU support without Docker as well. Configure and
build targets using the following commands:
```sh
./configure.py --backend=CUDA
bazel build \
--spawn_strategy=sandboxed \
--test_output=all \
//xla/...
```
For more details regarding
[hermetic CUDA you can check out this document.](https://github.com/google-ml-infra/rules_ml_toolchain/blob/main/gpu)
### Build XLA with CUDA/cuDNN Support Using the JAX CI/Release Container
XLA is a compiler used internally by JAX.
JAX is distributed via PyPI wheels.
The [JAX Continuous Integration documentation](https://github.com/jax-ml/jax/tree/main/ci#running-these-scripts-locally-on-your-machine)
explains how to build JAX wheels using
the [tensorflow/ml-build:latest](https://us-central1-docker.pkg.dev/tensorflow-sigs/tensorflow/ml-build) Docker container.
We can extend these instructions to build XLA targets within the JAX container
as well. This ensures that the XLA targets' build configuration is consistent
with the JAX/XLA build configuration, which can be useful if we want to
reproduce workload results using XLA tools that were originally created in JAX.
#### Build XLA Targets in the JAX CI Container
1. Clone the JAX repository and navigate to the 'jax' directory
```bash
git clone https://github.com/jax-ml/jax.git
cd jax
```
2. Start JAX CI/Release Docker container by running:
```bash
./ci/utilities/run_docker_container.sh
```
This will start a Docker container named 'jax'.
3. Build the jax-cuda-plugin target inside the container using:
```bash
docker exec jax ./ci/build_artifacts.sh jax-cuda-plugin
```
This will create the .jax_configure.bazelrc file with the required build
configuration, including CUDA/cuDNN support
4. Access an interactive shell inside the container:
```bash
docker exec -ti jax /bin/bash
```
You should now be in the `/jax` directory within the container
5. Build the XLA target with the following command, e.g.:
```bash
/usr/local/bin/bazel build \
--config=cuda_libraries_from_stubs \
--verbose_failures=true \
@xla//xla/tools/multihost_hlo_runner:hlo_runner_main
```
Optionally, you can overwrite `HERMETIC` envs, e.g.:
```bash
--repo_env=HERMETIC_CUDA_COMPUTE_CAPABILITIES="sm_90"
```
6. Copy the resulting artifacts to `/jax/dist` to access them from the host OS
if needed
```bash
cp bazel-bin/external/xla/xla/tools/multihost_hlo_runner/hlo_runner_main \
./dist/
```
7. Exit the interactive shell:
```bash
exit
```
## Windows
Building XLA on Windows natively is a CPU-only process, as CUDA is not
supported directly on Windows; you must use WSL2 if you need CUDA support. It
also requires specific environmental configurations, including a Bash shell, the
Clang compiler, and Visual Studio.
### Prerequisites
1. **Visual Studio:** You must install Visual Studio 2019 version 16.5 or newer
to set up a C++ toolchain (which provides the necessary system headers and
libraries).
2. **Bash Environment:** You must use a Bash shell (such as MSYS2 or Git Bash)
to build XLA on Windows.
3. **Clang Compiler:** XLA Windows builds use `clang-cl` rather than the
standard MSVC compiler. Ensure LLVM/Clang is installed.
4. **Python:** Python 3 must be installed and available in your system's
`PATH`.
5. **Developer Mode:** You must enable Developer Mode in Windows or run your
Bash shell as an Administrator. This is required because Bazel relies on
creating symlinks for the runfiles tree during the build process.
### Building from Source
To compile the CPU backend of XLA on Windows, run the following command from
your Bash terminal. We use the `xla_windows_x86_cpu_2022` config to
automatically set up the `clang-cl` toolchain.
Note that certain targets and tags must be explicitly excluded from the Windows
CPU build, such as GPU targets and features not yet supported on Windows:
```bash
# Note: Ensure your PATH includes Python and Bazel before running this
bazel build \
--config=xla_windows_x86_cpu_2022 \
--keep_going \
--build_tag_filters=-no_oss,-oss_excluded,-gpu,-no_windows,-windows_excluded \
-- //xla/... -//xla/hlo/experimental/... -//xla/python_api/... -//xla/python/...
```
third_party/xla/docs/contributing.md
Vendored
+287
View File
@@ -0,0 +1,287 @@
# Contributing to OpenXLA
Everyone can contribute to OpenXLA, and we value everyone’s contributions. There
are several ways to contribute, including:
* Answering questions on OpenXLA’s discussions forums (openxla-discuss)
* Improving or expanding OpenXLA’s documentation
* Contributing to OpenXLA’s code-base
* Contributing in any of the above ways to the broader ecosystem of libraries
built on OpenXLA
The OpenXLA project follows
[Google’s Open Source Community Guidelines](https://opensource.google/conduct/).
## Before you begin
### Sign the Contributor License Agreement
Contributions to this project must be accompanied by a
[Contributor License Agreement](https://cla.developers.google.com/about) (CLA).
You (or your employer) retain the copyright to your contribution; this simply
gives us permission to use and redistribute your contributions as part of the
project.
If you or your current employer have already signed the Google CLA (even if it
was for a different project), you probably don't need to do it again.
Visit <https://cla.developers.google.com/> to see your current agreements or to
sign a new one.
### Review the Code of Conduct
This project follows
[Tensorflow's Code of Conduct](https://github.com/tensorflow/tensorflow/blob/master/CODE_OF_CONDUCT.md).
## Contribution process
### Developer Guide
For a guide on how to setup a development environment for OpenXLA, including
getting code, building it, running tests and submitting changes, please refer to
the [Developer guide](./developer_guide.md).
### Contribution guide
The architecture of the compiler consists of the following components.
![img](./images/gpu_compiler.png)
#### Optimization passes
Optimization passes execute transformations on the HLO to enhance computational
efficiency. These transformations span from architecture-agnostic, high-level
improvements to hardware-specific adjustments (e.g., for NVIDIA GPUs).
##### What we generally accept:
* Passes that generalize across multiple workloads and demonstrate a clear and
significant positive impact on performance benchmarks.
##### What we generally reject:
* Passes that perform unique optimizations targeting specific models.
#### Fusion passes
Fusion is a critical optimization that combines multiple HLO operations into a
single kernel to reduce memory I/O and kernel launch overhead.
All fusion passes should be added to the fusion pipeline only, not before or
after. That also means that the pre-optimized HLO module should not contain
fusion instructions. If the fusion is formed early in the pipeline, it becomes a
barrier for the optimization passes. If the fusion is formed late, then we lose
an ability to select and tune the backend for the generated fusion.
Fusion into the custom calls, i.e. pattern-matching custom calls with the
producers/consumers and rewriting them into the new custom calls is not allowed.
In that case, it should be replaced with a proper fusion pass.
#### Backends & Autotuning
Backends for the unnested ops, e.g. custom calls and fusions, should implement
[CodegenBackend](https://github.com/openxla/xla/blob/main/xla/backends/autotuner/codegen_backend.h)
interface.
This interface is necessary to enable optimal backend selection, because it
provides the methods to include the parameters for the given HLO instructions
into the search space of the autotuner.
```
// Returns all supported configs for the given HLO instruction.
virtual absl::StatusOr<std::vector<std::unique_ptr<BackendConfig>>>
GetSupportedConfigs(const HloInstruction& instr);
// Returns a default config for the given HLO instruction.
virtual absl::StatusOr<std::unique_ptr<BackendConfig>> GetDefaultConfig(
const HloInstruction& instr);
```
#### Runtime
The end result of the XLA compilation pipeline is a thunk sequence that can be
serialized.
All of the new thunk types should be serializable, i.e. `GpuCompiler` or
`CpuCompiler` should be able to compile the program, serialize it, so that later
the XLA runner could load and execute the program. That means that there should
be no pointers to `HloInstruction` or to other parts of the compiler or the
`StreamExecutor`.
### Code standards
* *Coding style*: We follow [Google's code style guide](https://google.github.io/styleguide/).
Specifically see the [C/C++](https://google.github.io/styleguide/cppguide.html) and [Python](https://google.github.io/styleguide/pyguide.html) guides. All
code submitted must strictly conform to these style guides.
* *Compact changes*: We follow
[Google's engineering practices](https://google.github.io/eng-practices/).
In particular, please observe the
[guide on writing compact changes](https://google.github.io/eng-practices/review/developer/small-cls.html).
Doing so will greatly increase the speed at which you can get your code
merged due to improve reviewability, and reducing the likelihood of
unintentional side effects of change. Even if you have a large change, there
are many strategies for breaking it up into more incremental changes. If
your PR is too large, it will receive an automated comment asking you to
break it down into smaller PRs.
* *Test Coverage*: All changes should include appropriate unit tests. Unit
tests should not be dependent on specific hardware (CPU, GPU, etc.) timings,
and should make liberal use of mocks and fakes in order to make
deterministic and focused tests. Changes seeking to extend existing code
that’s currently hard to test should make appropriate improvements to
testability.
All changes should include appropriate benchmark results as well in the
change title to ensure the benefits are clearly understood.
* *Feature Flags*: All somewhat complicated new features should be guarded
with a flag first (e.g., via `DebugOptions`). This allows for easy rollback
of the flag flip if problems arise, and affected users can temporarily
set the flag themselves before a rollback is performed.
* When in doubt as to conventions within the code, it is always a good idea to
examine pre-existing code and to try to follow the patterns already in place
in OpenXLA.
### Review Process
All submissions, including submissions by project members, require review. We
use GitHub pull requests for this purpose. Consult
[GitHub Help](https://help.github.com/articles/about-pull-requests/) for more
information on using pull requests.
* Code must follow all standards listed above prior to review. These are not
optional and it is critical that the submitter ensure their code conforms
before requesting review in order to assure timely acceptance of changes.
* *All tests and additional checks on GitHub must pass*. If you find that a
test is broken and the issue is not related to your build environment or
otherwise your changes, please contact the maintainers.
* Avoid scope creep during the review process. This is the
responsibility of both the submitter and the reviewer. If a change starts to
get too large, consider breaking it up into multiple changes.
* After a change is approved on GitHub but before it is merged, it will
undergo internal testing that uses code
internal to Google and other hardware vendors. This can potentially add extra
steps to the review process if there are failures on internal tests that our
public CI doesn't catch. The Googler reviewing your change will communicate
any internal test failures and describe what needs to be fixed. The overall
state of the internal checks is visible in the checks list on GitHub:
- *import/copybara — Change imported to the internal review system*:
Your PR has been imported in Google's internal system and checks are
running.
- *import/copybara — An error happened while migrating the change*: Your PR
could not be imported into Google's internal system. This very rarely
happens. If you see this state please ping your reviewer.
- *feedback/copybara — Google internal checks PASS for runs with create time...*:
All internal checks pass. Your PR should be merged soon.
- *feedback/copybara — Google internal checks FAILED for runs with create time ...*:
Some internal checks failed. A Google engineer will soon post a comment with
more details. If you don't get any info about the failures within a day
please ping your reviewer.
## Frequently asked questions (FAQ)
### "This infrastructure change is not related to my PR. Why should I do it?"
The XLA team doesn't have a dedicated infrastructure team, so it's up to us all
to build helper libraries and avoid technical debt. We consider it to be a
regular part of making changes to XLA, and everyone is expected to participate.
We generally build infrastructure as needed when writing code.
XLA reviewers may ask you to build some infrastructure (or otherwise make a
large change to a PR) along with a PR that you've written. This request may seem
unnecessary or orthogonal to the change you're trying to make. This is likely
because of a mismatch between your expectations about how much infra you need to
build and your reviewer's expectations for the same.
A mismatch in expectations is okay! That's expected when you're new to a project
(and it sometimes even happens to us old hats). It's likely that projects you've
worked on in the past have different expectations. That's also okay and
expected! It doesn't mean either one of these projects has the wrong approach;
they're just different. We invite you to take infra requests alongside all other
review comments as an opportunity to learn what we expect on this project.
### "Can I address your comment in a future PR?"
A frequent question with respect to infrastructure requests (or other large
requests) in PRs is whether or not the change must be made in the original PR,
or whether it can be done as a follow-up in a future PR.
In general, XLA does not allow PR authors to address review comments with a
follow-up PR. When a reviewer decides that something needs to be addressed in a
given PR, we generally expect authors to address it in that PR, even if what's
requested is a large change. This standard applies externally and also
internally within Google.
There are a few reasons that XLA takes this approach.
* *Trust:* Having earned the reviewer's trust is a key component. In an
open-source project, contributors can appear or disappear at will. After we
approve a PR, reviewers have no way to ensure that any promised follow-ups
actually get done.
* *Impact on other developers:* If you have sent a PR touching a particular
part of XLA, there's a good chance other people are looking at the same
part. If we accept technical debt in your PR, then everyone who's looking at
this file will be impacted by this debt until the follow-up is submitted.
* *Reviewer bandwidth:* Deferring a change to a follow-up imposes multiple
costs on our already overloaded reviewers. Reviewers will probably forget
what the first PR was about while waiting for the follow-up, making the next
review more difficult. Also, reviewers will have to keep track of expected
follow-ups, making sure that they actually happen. If the change can be made
such that it is truly orthogonal to the original PR so that some other
reviewer could review it, bandwidth would be less of a problem. In our
experience, this is rarely the case.
### What is the turn around time for code review and merge?
The OpenXLA project has a large contributor base. While we strive for quick
reviews and code merges, there are delays due to peaks of work.
In order to empower our partner teams to contribute high quality features and
fixes to the codebase, and to get quicker reviews and merges, we have
established the co-maintainer program. In this program, selected trusted partner
contributors ensure that the PRs submitted by those partners meet OpenXLA
quality requirements as specified in this contributing guide.
If co-maintainers from external partner teams have approved a PR, the Google XLA
team commits to reviewing and merging (if approved) within a controlled SLO
(**t<sub>Google Review</sub>** in the diagram below). If the PR is not approved,
the Google XLA team will provide a documented justification and where available
a reproducer of the failure.
At this moment the list of co-maintainers from our partner teams is:
* [@Tixxx](https://github.com/Tixxx)
* [@sergachev](https://github.com/sergachev)
* [@jreiffers](https://github.com/jreiffers)
* [@ezhulenev](https://github.com/ezhulenev)
* [@mminutoli](https://github.com/mminutoli)
* [@pemeliya](https://github.com/pemeliya)
![img](./images/comaintainers.png)
| Percentile | Target for **t<sub>Google Review</sub>** |
| --- | --- |
| 50 percentile | 3 business days |
| 80 percentile | 4 business days |
| 90 percentile | 5 business days |
| 95 percentile | 6 business days |
When Google reviews the PR, it may fail due to an internal test. In this case,
Google will try to generate a reproducer and share it with the partner. In other
cases, the review may fail for technical, architectural or stylish reasons -
Google will share the feedback with the author. We expect that if the PR is
rejected in the first pass, we will enter a tight loop iteration between the
partner team and Google (“Iterative review”) to bring the PR to the desired
state.
The OpenXLA project is committed to strengthening its co-maintainer program and
adding more contributors to this list. We plan to review on a monthly basis the
target turnaround times and report back to the participating teams.
third_party/xla/docs/copybara.md
Vendored
+29
View File
@@ -0,0 +1,29 @@
# Copybara quirks
The purpose of this document is to describe oddities you might see while
contributing due to the tool that manages copying source back and forth from
Google's internal repository. This tool is called [Copybara](https://github.com/google/copybara).
## Internal source of truth
Because the source of truth for the code in this repository is Google's internal
repo, Copybara does transformations to the code whenever the code is imported
and exported. This means that sometimes seemingly normal changes can break
internally in surprising ways.
## PR merge status and diff inconsistencies
Since the source of truth is internal, PRs are not merged directly, they are
imported to the Google internal repo where they undergo additional testing,
and then that internal change is submitted, and attributed to the PR author.
Because of the transformations that Copybara applies, there's no guarantee that
the diff will be identical (for example, Copybara applies formatting on import).
For this reason, Copybara won't mark the PR as merged, it will close the PR and
separately apply a commit that should map very closely to the PR.
## Dependency on TSL by copy
As implemented currently, to prevent any temporary broken commits, XLA
depends on TSL not by downloading a copy by using Bazel's `http_archive`, but by
having Copybara copy TSL into XLA's `third_party` directory.
third_party/xla/docs/custom_call.md
Vendored
+448
View File
@@ -0,0 +1,448 @@
# XLA Custom Calls
This document describes how to write and use XLA custom calls using XLA FFI
library. Custom call is a mechanism to describe an external "operation" in the
HLO module to the XLA compiler (at compile time), and XLA FFI is a mechanism to
register implementation of such operations with XLA (at run time). FFI stands
for "foreign function interface" and it is a set of C APIs that define a binary
interface (ABI) for XLA to call into external code written in other programming
languages. XLA provides header-only bindings for XLA FFI written in C++, which
hides all the low level details of underlying C APIs from the end user.
> **Caution:** The custom-call API/ABI uses PJRT-style versioning (major, minor),
> however at this point it is still experimental and can be broken at any time.
> Once API/ABI is finalized we intend to provide stability guarantees
> similar to PJRT.
> **Caution** The HLO-visible names of functions registered with the custom-call
> macros API do not respect C++ namespaces. As a result, accidental collisions
> from functions registered by different libraries are entirely possible! The
> API will reject such duplicate registrations, but to avoid issues in large
> projects the safest option is to either fully namespace-qualify all references
> to the functions in both the `XLA_REGISTER_CUSTOM_CALL` registration macros
> and custom call target references or to use C-style namespacing directly in
> the function name.
## JAX + XLA Custom Calls
See [JAX documentation](https://jax.readthedocs.io/en/latest/ffi.html) for
end to end examples of integrating custom calls and XLA FFI with JAX.
## XLA FFI Binding
XLA FFI binding is a compile-time specification of the custom call signature:
custom call arguments, attributes and their types, and additional parameters
passed via the execution context (i.e., gpu stream for GPU backend). XLA FFI
binding can be bound to any C++ callable (function pointer, lambda, etc.) with
compatible `operator()` signature. Constructed handler decodes XLA FFI call
frame (defined by the stable C API), type check all parameters, and forward
decoded results to the user-defined callback.
XLA FFI binding heavily relies on template metaprogramming to be be able to
compile constructed handler to the most efficient machine code. Run time
overheads are in order of a couple of nanoseconds for each custom call
parameter.
XLA FFI customization points implemented as template specializations, and
users can define how to decode their custom types, i.e., it is possible
to define custom decoding for user-defined `enum class` types.
### Returning Errors From Custom Calls
Custom call implementations must return `xla::ffi::Error` value to signal
success or error to XLA runtime. It is similar to `absl::Status`, and has
the same set of error codes. We do not use `absl::Status` because it does
not have a stable ABI and it would be unsafe to pass it between dynamically
loaded custom call library, and XLA itself.
```c++
// Handler that always returns an error.
auto always_error = Ffi::Bind().To(
[]() { return Error(ErrorCode::kInternal, "Oops!"); });
// Handler that always returns a success.
auto always_success = Ffi::Bind().To(
[]() { return Error::Success(); });
```
### Buffer Arguments And Results
XLA uses destination passing style for results: custom calls (or any other XLA
operations for that matter) do not allocate memory for results, and instead
write into destinations passed by XLA runtime. XLA uses static buffer
assignment, and allocates buffers for all values based on their live ranges at
compile time.
Results passed to FFI handlers wrapped into a `Result<T>` template, that
has a pointer-like semantics: `operator->` gives access to the underlying
parameter.
`AnyBuffer` arguments and results gives access to custom call buffer parameters
of any data type. This is useful when custom call has a generic implementation
that works for multiple data types, and custom call implementation does run time
dispatching based on data type. `AnyBuffer` gives access to the buffer data
type, dimensions, and a pointer to the buffer itself.
```mlir
%0 = "stablehlo.custom_call"(%arg0) {
call_target_name = "foo",
api_version = 4 : i32
} : (tensor<2x2xf32>) -> tensor<2x2xf32>
```
```c++
// Buffers of any number of dimensions and data type.
auto handler = Ffi::Bind().Arg<AnyBuffer>().Ret<AnyBuffer>().To(
[](AnyBuffer arg, Result<AnyBuffer> res) -> Error {
void* arg_data = arg.untyped_data();
void* res_data = res->untyped_data();
return Error::Success();
});
```
### Constrained Buffer Arguments And Results
`Buffer` allows to add constraints on the buffer data type and number of
dimensions, and they will be automatically checked by the handler and return
an error to XLA runtime, if run time arguments do not match the FFI handler
signature.
```c++
// Buffers of any number of dimensions and F32 data type.
auto handler = Ffi::Bind().Arg<Buffer<F32>>().Ret<Buffer<F32>>().To(
[](Buffer<F32> arg, Result<Buffer<F32>> res) -> Error {
float* arg_data = arg.typed_data();
float* res_data = res->typed_data();
return Error::Success();
});
```
```c++
// Buffers of number of dimensions 2 and F32 data type.
auto handler = Ffi::Bind().Arg<BufferR2<F32>>().Ret<BufferR2<F32>>().To(
[](BufferR2<F32> arg, Result<BufferR2<F32>> res) -> Error {
float* arg_data = arg.typed_data();
float* res_data = res->typed_data();
return Error::Success();
});
```
### Variadic Arguments And Results
If the number of arguments and result can be different in different instances of
a custom call, they can be decoded at run time using `RemainingArgs` and
`RemainingRets`.
```
auto handler = Ffi::Bind().RemainingArgs().RemainingRets().To(
[](RemainingArgs args, RemainingRets results) -> Error {
ErrorOr<AnyBuffer> arg = args.get<AnyBuffer>(0);
ErrorOr<Result<AnyBuffer>> res = results.get<AnyBuffer>(0);
if (!arg.has_value()) {
return Error(ErrorCode::kInternal, arg.error());
}
if (!res.has_value()) {
return Error(ErrorCode::kInternal, res.error());
}
return Error::Success();
});
```
Variadic arguments and results can be declared after regular arguments and
results, however binding regular arguments and results after variadic one is
illegal.
```c++
auto handler =
Ffi::Bind()
.Arg<AnyBuffer>()
.RemainingArgs()
.Ret<AnyBuffer>()
.RemainingRets()
.To([](AnyBuffer arg, RemainingArgs args, AnyBuffer ret,
RemainingRets results) -> Error { return Error::Success(); });
```
### Attributes
XLA FFI supports automatic decoding of `mlir::DictionaryAttr` passed as a
`custom_call` `backend_config` into FFI handler arguments.
Note: See [stablehlo RFC](https://github.com/openxla/stablehlo/blob/main/rfcs/20240312-standardize-customcallop.md)
for details, and `stablehlo.custom_call` operation specification.
```mlir
%0 = "stablehlo.custom_call"(%arg0) {
call_target_name = "foo",
backend_config= {
i32 = 42 : i32,
str = "string"
},
api_version = 4 : i32
} : (tensor<f32>) -> tensor<f32>
```
In this example custom call has a single buffer argument and two attributes, and
XLA FFI can automatically decode them and pass to the user-defined callable.
```c++
auto handler = Ffi::Bind()
.Arg<BufferR0<F32>>()
.Attr<int32_t>("i32")
.Attr<std::string_view>("str")
.To([](BufferR0<F32> buffer, int32_t i32, std::string_view str) {
return Error::Success();
});
```
### User-Defined Enum Attributes
XLA FFI can automatically decode integral MLIR attributes into user-defined
enums. Enum class must have the same underlying integral type, and decoding
has to be explicitly registered with XLA FFI.
```mlir
%0 = "stablehlo.custom_call"(%arg0) {
call_target_name = "foo",
backend_config= {
command = 0 : i32
},
api_version = 4 : i32
} : (tensor<f32>) -> tensor<f32>
```
```c++
enum class Command : int32_t {
kAdd = 0,
kMul = 1,
};
XLA_FFI_REGISTER_ENUM_ATTR_DECODING(Command);
auto handler = Ffi::Bind().Attr<Command>("command").To(
[](Command command) -> Error { return Error::Success(); });
```
### Binding All Custom Call Attributes
It is possible to get access to all custom call attributes as a dictionary
and lazily decode only the attributes that are needed at run time.
```c++
auto handler = Ffi::Bind().Attrs().To([](Dictionary attrs) -> Error {
ErrorOr<int32_t> i32 = attrs.get<int32_t>("i32");
return Error::Success();
});
```
### User-defined Struct Attributes
XLA FFI can decode dictionary attributes into user-defined structs.
```mlir
%0 = "stablehlo.custom_call"(%arg0) {
call_target_name = "foo",
backend_config= {
range = { lo = 0 : i64, hi = 42 : i64 }
},
api_version = 4 : i32
} : (tensor<f32>) -> tensor<f32>
```
In example above `range` is an `mlir::DictionaryAttr` attribute, and instead
of accessing dictionary fields by name, it can be automatically decoded as
a C++ struct. Decoding has to be explicitly registered with a
`XLA_FFI_REGISTER_STRUCT_ATTR_DECODING` macro (behind the scene it defines
a template specialization in `::xla::ffi` namespace, thus macro must be added to
the global namespace).
```c++
struct Range {
int64_t lo;
int64_t hi;
};
XLA_FFI_REGISTER_STRUCT_ATTR_DECODING(Range, StructMember<int64_t>("lo"),
StructMember<int64_t>("hi"));
auto handler = Ffi::Bind().Attr<Range>("range").To([](Range range) -> Error{
return Error::Success();
});
```
Custom attributes can be loaded from a dictionary, just like any other
attribute. In example below, all custom call attributes decoded as a
`Dictionary`, and a `range` can be accessed by name.
```c++
auto handler = Ffi::Bind().Attrs().To([](Dictionary attrs) -> Error {
ErrorOr<Range> range = attrs.get<Range>("range");
return Error::Success();
});
```
## Create a custom call on CPU
You can create an HLO instruction that represents a custom call via XLA's client
API. For example, the following code uses a custom call to compute `A[i] = B[i %
128]+ C[i]` on the CPU. (Of course you could &ndash; and should! &ndash; do this
with regular HLO.)
```c++
#include "xla/client/xla_builder.h"
#include "xla/service/custom_call_target_registry.h"
void do_it() {
xla::XlaBuilder b("do_it");
xla::XlaOp param0 =
xla::Parameter(&b, 0, xla::ShapeUtil::MakeShape(xla::F32, {128}), "p0");
xla::XlaOp param1 =
xla::Parameter(&b, 1, xla::ShapeUtil::MakeShape(xla::F32, {2048}), "p1");
xla::XlaOp custom_call =
xla::CustomCall(&b, "do_custom_call", /*operands=*/{param0, param1},
/*shape=*/xla::ShapeUtil::MakeShape(xla::F32, {2048}),
/*opaque=*/"", /*has_side_effect=*/false,
/*output_operand_aliasing=*/{}, /*literal=*/nullptr,
/*schedule=*/CustomCallSchedule::SCHEDULE_NONE,
/*api_version=*/CustomCallApiVersion::API_VERSION_TYPED_FFI);
}
// Constrain custom call arguments to 1-dimensional buffers of F32 data type.
using BufferF32 = xla::ffi::BufferR1<xla::ffi::DataType::F32>;
// Implement a custom call as a C++ function. Note that we can use `Buffer` type
// defined by XLA FFI that gives us access to buffer data type and shape.
xla::ffi::Error do_custom_call(BufferF32 in0, BufferF32 in1,
xla::ffi::Result<BufferF32> out) {
size_t d0 = in0.dimensions[0];
size_t d1 = in1.dimensions[0];
// Check that dimensions are compatible.
assert(out->dimensions[0] == d1 && "unexpected dimensions");
for (size_t i = 0; i < d1; ++i) {
out->data[i] = in0.data[i % d0] + in1.data[i];
}
}
// Explicitly define an XLA FFI handler signature and bind it to the
// `do_custom_call` implementation. XLA FFI handler can automatically infer
// type signature from the custom call function, but it relies on magical
// template metaprogramming an explicit binding provides and extra level of
// type checking and clearly states custom call author intentions.
XLA_FFI_DEFINE_HANDLER(handler, do_custom_call,
ffi::Ffi::Bind()
.Arg<Buffer>()
.Arg<Buffer>()
.Ret<Buffer>());
// Registers `handler` with and XLA FFI on a "Host" platform.
XLA_FFI_REGISTER_HANDLER(xla::ffi::GetXlaFfiApi(), "do_custom_call",
"Host", handler);
```
## Create a custom call on GPU
The GPU custom call registration with XLA FFI is almost identical, the only
difference is that for GPU you need to ask for an underlying platform stream
(CUDA or ROCM stream) to be able to launch kernel on device. Here is a CUDA
example that does the same computation (`A[i] = B[i % 128] + C[i]`) as the CPU
code above.
```c++
void do_it() { /* same implementation as above */ }
__global__ custom_call_kernel(const float* in0, const float* in1, float* out) {
size_t idx = blockIdx.x * blockDim.x + threadIdx.x;
out[idx] = in0[idx % 128] + in1[idx];
}
void do_custom_call(CUstream stream, BufferF32 in0, BufferF32 in1,
xla::ffi::Result<BufferF32> out) {
size_t d0 = in0.dimensions[0];
size_t d1 = in1.dimensions[0];
size_t d2 = out->dimensions[0];
assert(d0 == 128 && d1 == 2048 && d2 == 2048 && "unexpected dimensions");
const int64_t block_dim = 64;
const int64_t grid_dim = 2048 / block_dim;
custom_call_kernel<<<grid_dim, block_dim, 0, stream>>>(
in0.data, in1.data, out->data);
}
XLA_FFI_DEFINE_HANDLER(handler, do_custom_call,
ffi::Ffi::Bind()
.Ctx<xla::ffi::PlatformStream<CUstream>>()
.Arg<BufferF32>()
.Arg<BufferF32>()
.Ret<BufferF32>());
XLA_FFI_REGISTER_HANDLER(xla::ffi::GetXlaFfiApi(), "do_custom_call",
"CUDA", handler);
```
Notice first that the GPU custom call function *is still a function executed on
the CPU*. The `do_custom_call` CPU function is responsible for enqueueing work
on the GPU. Here it launches a CUDA kernel, but it could also do something else,
like call cuBLAS.
Arguments and results also live on the host, and data member contains a pointer
to device (i.e. GPU) memory. Buffers passed to custom call handler have the
shape of the underlying device buffers, so the custom call can compute kernel
launch parameters from them.
## Passing tuples to custom calls
Consider the following custom call.
```c++
using xla::ShapeUtil;
using xla::F32;
Shape p0_shape = ShapeUtil::MakeTuple({
ShapeUtil::MakeShape(F32, {32}),
ShapeUtil::MakeTuple({
ShapeUtil::MakeShape(F32, {64}),
ShapeUtil::MakeShape(F32, {128}),
}),
ShapeUtil::MakeShape(F32, {256}),
});
xla::XlaOp p0 = xla::Parameter(0, p0_shape, "p0");
Shape out_shape = ShapeUtil::MakeTuple({
ShapeUtil::MakeShape(F32, {512}),
ShapeUtil::MakeShape(F32, {1024}),
});
xla::CustomCall(&b, "do_custom_call", /*operands=*/{p0}, out_shape, ...);
```
On both CPU and GPU, a tuple is represented in memory as an array of pointers.
When XLA calls custom calls with tuple arguments or results it flattens them and
passes as regular buffer arguments or results.
### Tuple outputs as temp buffers
Tuple inputs to custom calls are a convenience, but they aren't strictly
necessary. If we didn't support tuple inputs to custom calls, you could always
unpack the tuples using get-tuple-element before passing them to the custom
call.
On the other hand, tuple *outputs* do let you do things you couldn't otherwise.
The obvious reason to have tuple outputs is that tuple outputs are how a custom
call (or any other XLA op) returns multiple independent arrays.
But less obviously, a tuple output is also a way to give your custom call temp
memory. Yes, an *output* can represent a temp buffer. Consider, an output buffer
has the property that the op can write to it, and it can read from it after it's
been written to. That's exactly what you want from a temp buffer.
In the example above, suppose we wanted to use the `F32[1024]` as a temp buffer.
Then we'd write the HLO just as above, and we'd simply never read tuple index 1
of the custom call's output.
third_party/xla/docs/determinism.md
Vendored
+21
View File
@@ -0,0 +1,21 @@
# Determinism (GPU)
## Compilation
XLA compilation is deterministic if
[persisted autotuning](./persisted_autotuning) is used to perform autotuning
once and avoid it in subsequent compilations. Otherwise due to fluctuations in
measurements different kernels can be picked as the fastest ones in different
compilation runs.
`--xla_gpu_require_complete_aot_autotune_results` can be used to ensure that no
autotuning happens on repeated compilations - they either reuse compatible
results of previous runs or fail.
## Execution
Programs compiled by XLA can be non-deterministic on operations like scatter,
select-and-scatter, GEMMs, convolutions, multi-headed attention. The flag
`--xla_gpu_exclude_nondeterministic_ops` switches these operations to
deterministic and potentially slower implementations and makes compilation fail
on select-and-scatter which does not have a deterministic implementation.
third_party/xla/docs/developer_environment.md
Vendored
+70
View File
@@ -0,0 +1,70 @@
# Setting Up Developer Environment
## Setting up LSP with clangd
### Background
Editors such as Emacs, Vim, or VS Code support features like code navigation,
code completion, inline compiler error messages, and others, through
[LSP](https://en.wikipedia.org/wiki/Language_Server_Protocol), the Language
Server Protocol. A common language server with LSP support is
[clangd](https://clangd.llvm.org), which relies on the presence of
`compile_commands.json`, a JSON file with a record of the compile commands for
each file in a project.
### How do I generate `compile_commands.json` for XLA source code?
Use the
[build_tools/lint/generate_compile_commands.py](https://github.com/openxla/xla/blob/main/build_tools/lint/generate_compile_commands.py)
script. The following invocation from XLA repo root generates a
`compile_commands.json` file in place: `bazel aquery "mnemonic(CppCompile,
//xla/...)" --output=jsonproto | python3
build_tools/lint/generate_compile_commands.py`
## Build Cleaner
XLA CI inside Google runs additional checks to verify that all targets correctly
list all dependencies in the BUILD files, which are not enabled by default in
OSS Bazel CI. Running following commands before sending PR to XLA team will
massively speedup the time it takes to merge the PR as otherwise you will need
some Googler to make this fixes for you internally, or a few more rounds of PR
reviews to fix them based on the Googlers feedback.
### Layering Check
Building with `--features=layering_check` makes sure that you don't accidentally
include a header via transitive dependency without listing it in your target
dependencies
### Remove unused dependencies from BUILD files
Install
[Buildozer](https://github.com/bazelbuild/buildtools/blob/main/buildozer/README.md)
tool:
```
sudo curl -fsSL -o /usr/bin/buildozer https://github.com/bazelbuild/buildtools/releases/download/6.0.0/buildozer-linux-amd64
sudo chmod 755 /usr/bin/buildozer
```
Install [Bant](https://github.com/hzeller/bant) tool:
```
# To some writable directory that does not require root access
bazel build -c opt //bant && install -D --strip bazel-bin/bant/bant ~/bin/bant
# For a system directory that requires root-access
sudo install -D --strip bazel-bin/bant/bant /usr/local/bin/bant
```
Use `bant` to generate buildozer commands to remove unused deps:
```
bant dwyu //xla/core/collectives:symmetric_memory
```
if you feel lucky, you can execute them directly:
```
. <(bant dwyu //xla/core/collectives:symmetric_memory)
```
third_party/xla/docs/developer_guide.md
Vendored
+164
View File
@@ -0,0 +1,164 @@
# XLA developer guide
This guide shows you how to get started developing the XLA project.
Before you begin, complete the following prerequisites:
1. Go to [Contributing page](contributing.md) and review the contribution
process.
2. If you haven't already done so, sign the
[Contributor License Agreement](https://cla.developers.google.com/about).
3. Install or configure the following dependencies:
- A [GitHub](https://github.com/) account
- [Docker](https://www.docker.com/)
Then follow the steps below to get the source code, set up an environment, build
the repository, and create a pull request.
## Get the code
1. Create a fork of the [XLA repository](https://github.com/openxla/xla).
2. Clone your fork of the repo, replacing `{USER}` with your GitHub username:
```sh
git clone https://github.com/{USER}/xla.git
```
3. Change into the `xla` directory: `cd xla`
4. Configure the remote upstream repo:
```sh
git remote add upstream https://github.com/openxla/xla.git
```
## Set up an environment
1. Install [Bazel](https://bazel.build/install).
To build XLA, you must have Bazel installed. The recommended way to install
Bazel is using [Bazelisk](https://github.com/bazelbuild/bazelisk#readme),
which automatically downloads the correct Bazel version for XLA. If Bazelisk
is unavailable, you can [install Bazel](https://bazel.build/install)
manually.
2. Create and run the
[ml-build](https://us-docker.pkg.dev/ml-oss-artifacts-published/ml-public-container/ml-build)
Docker container.
To set up a Docker container for building XLA with support for both CPU and
GPU, run the following command:
```sh
docker run -itd --rm \
--name xla \
-w /xla \
-v $PWD:/xla \
us-docker.pkg.dev/ml-oss-artifacts-published/ml-public-container/ml-build:latest \
bash
```
If building with GPU/CUDA support, add `--gpus all` to grant the container
access to all available GPUs. This enables automatic detection of CUDA
compute capabilities.
## Build
Configure for CPU:
```sh
docker exec xla ./configure.py --backend=CPU
```
Configure for GPU:
```sh
docker exec xla ./configure.py --backend=CUDA
```
CUDA compute capabilities will be detected automatically by running
`nvidia-smi`. If GPUs are not available during the build, you must specify
the compute capabilities manually. For example:
```sh
# Automatically detects compute capabilities (requires GPUs)
./configure.py --backend=CUDA
# Manually specify compute capabilities (for builds without GPUs)
./configure.py --backend=CUDA --cuda_compute_capabilities="9.0"
```
Build:
```sh
docker exec xla bazel build \
--spawn_strategy=sandboxed \
--test_output=all \
//xla/...
```
**Note:** You can build XLA on a machine without GPUs. In that case:
- Do **not** use `--gpus all` flag when starting the Docker container.
- During `./configure.py`, manually specify the CUDA compute capabilities
using the `--cuda_compute_capabilities` flag.
**Note:** Thanks to hermetic CUDA rules, you don't need to build XLA inside a
Docker container. You can build XLA for GPU directly on your machine - even if
it doesn't have a GPU or the NVIDIA driver installed.
```sh
# Automatically detects compute capabilities (requires GPUs)
./configure.py --backend=CUDA
# Manually specify compute capabilities (for builds without GPUs)
./configure.py --backend=CUDA --cuda_compute_capabilities="9.0"
bazel build \
--spawn_strategy=sandboxed \
--test_output=all \
//xla/...
```
Your first build will take quite a while because it has to build the entire
stack, including XLA, MLIR, and StableHLO.
To learn more about building XLA, see [Build from source](build_from_source.md).
## Create a pull request
When you're ready to send changes for review, create a
[pull request](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-pull-requests).
To learn about the XLA code review philosophy, see
[Review Process](contributing.md#review-process).
## Static Analysis (Clang-Tidy)
To maintain code quality, XLA uses `clang-tidy` for static analysis and include
verification.
### How to Run
There are two ways to execute checks. Running it against specific targets can be
done with:
```sh
bazel build --config=clang-tidy //path/to:target1 //path/to:target2
```
There is a helper script that is also used in CI workflows that runs it against
git diff from feature branch against the upstream main.
```sh
# Make sure the main is updated.
git fetch origin main
bazel run //build_tools/ci:run_clang_tidy
```
The helper script also allows to automagically fix clang-tidy errors where
possible.
```sh
bazel run //build_tools/ci:run_clang_tidy -- --fix
```
third_party/xla/docs/developing_new_backend.md
Vendored
+76
View File
@@ -0,0 +1,76 @@
# Developing a new backend for XLA
This guide is for system engineers who want XLA to output programs that target
their hardware efficiently. The guide is not step-by-step and assumes knowledge
of [LLVM](http://llvm.org), [Bazel](https://bazel.build/), and XLA.
XLA provides an abstract interface that a new architecture or accelerator can
implement to create a backend to run ML programs output by XLA. Retargeting XLA
should be significantly simpler and more scalable than implementing every
existing op from a frontend framework such as PyTorch or TensorFlow for new
hardware.
Most implementations will fall into one of the following scenarios:
1. Existing CPU architecture not yet officially supported by XLA, with or
without an existing [LLVM](http://llvm.org) backend.
2. Non-CPU-like hardware with an existing LLVM backend.
3. Non-CPU-like hardware without an existing LLVM backend.
> **Note:** An LLVM backend can mean either one of the officially released LLVM
> backends or a custom LLVM backend developed in-house.
## Scenario 1: Existing CPU architecture not yet officially supported by XLA
In this scenario, start by looking at the existing
[XLA CPU backend](https://github.com/openxla/xla/tree/main/xla/service/cpu). XLA
makes it easy to target different CPUs by using LLVM, because the main
difference between XLA backends for CPUs is the code generated by LLVM.
If the hardware vendor has an LLVM backend for their hardware, it is simple to
link the backend with the LLVM built with XLA. In JIT mode, the XLA CPU backend
emits code for the host CPU. For ahead-of-time compilation,
[`xla::AotCompilationOptions`](https://github.com/openxla/xla/tree/main/xla/service/compiler.h)
can provide an LLVM triple to configure the target architecture.
If there is no existing LLVM backend, but another kind of code generator exists,
it should be possible to reuse most of the existing CPU backend.
## Scenario 2: Non-CPU-like hardware with an existing LLVM backend
It is possible to model a new
[`xla::Compiler`](https://github.com/openxla/xla/tree/main/xla/service/compiler.h)
implementation on the existing
[`xla::CPUCompiler`](https://github.com/openxla/xla/tree/main/xla/service/cpu/cpu_compiler.cc)
and
[`xla::GPUCompiler`](https://github.com/openxla/xla/tree/main/xla/service/gpu/nvptx_compiler.cc)
classes, since these already emit LLVM IR. Depending on the nature of the
hardware, it is possible that many aspects of the LLVM IR generation will have
to be changed, but a lot of code can be shared with the existing backends.
A good example to follow is the
[GPU backend](https://github.com/openxla/xla/tree/main/xla/service/gpu/)
of XLA. The GPU backend targets a non-CPU-like ISA, and therefore some aspects
of its code generation are unique to the GPU domain. Other kinds of hardware,
e.g. DSPs like Hexagon (which has an upstream LLVM backend), can reuse parts of
the LLVM IR emission logic, but other parts will be unique.
## Scenario 3: Non-CPU-like hardware without an existing LLVM backend
If it is not possible to utilize LLVM, then the best option is to implement a
new backend for XLA for the desired hardware. This option requires the most
effort. The classes that need to be implemented are as follows:
* [`StreamExecutor`](https://github.com/openxla/xla/tree/main/xla/stream_executor/stream_executor.h):
For many devices not all methods of `StreamExecutor` are needed. See
existing `StreamExecutor` implementations for details.
* [`xla::Compiler`](https://github.com/openxla/xla/tree/main/xla/service/compiler.h):
This class encapsulates the compilation of an HLO computation into an
`xla::Executable`.
* [`xla::Executable`](https://github.com/openxla/xla/tree/main/xla/service/executable.h):
This class is used to launch a compiled computation on the platform.
* [`xla::TransferManager`](https://github.com/openxla/xla/tree/main/xla/service/transfer_manager.h):
This class enables backends to provide platform-specific mechanisms for
constructing XLA literal data from given device memory handles. In other
words, it helps encapsulate the transfer of data from the host to the device
and back.
third_party/xla/docs/effort_levels.md
Vendored
+80
View File
@@ -0,0 +1,80 @@
# Effort Levels
XLA provides options to control the amount of effort the compiler will expend to
* optimize for runtime performance, and
* make the program "fit in memory" (which has a platform-dependent meaning)
## Optimization Level
Similar to the -O flags in gcc or clang, this field allows the user to influence
how much work the compiler does in optimizing for execution time. It can be set
via the
[optimization_level](https://github.com/openxla/xla/blob/f4d624b6811c28925c3006f5b779f1149b3b39ac/xla/pjrt/proto/compile_options.proto#L71)
field of the ExecutableBuildOptionsProto message, or the
[optimization_level](https://github.com/openxla/xla/blob/f4d624b6811c28925c3006f5b779f1149b3b39ac/xla/xla.proto#L1580)
field of the ExecutionOptions message.
Lower optimization levels will cause various HLO passes to behave differently,
typically doing less work, or may disable certain HLO passes entirely. The
optimization level may also influence the compiler backend, such that the exact
effect of this field has a dependence on the target platform. However, as a
general guideline, the following table describes the expected overall effect of
each value:
| Level | Use Case |
| :-------- | :---------------------------------------------------------------------- |
| EFFORT_O0 | Fastest compilation, slowest runtime |
| EFFORT_O1 | Faster compilation with reasonable runtime |
| EFFORT_O2 | Strongly prioritize runtime (suitable default for production workloads) |
| EFFORT_O3 | Expensive or experimental optimizations |
### Use in XLA:GPU
In XLA:GPU, there are several passes that we disable by default because they
significantly increase compilation time by increasing the HLO size. For
convenience, we consolidate them under the optimization level option, such that
setting optimization_level to O1 or above will lead to the following behavior:
* Collectives commonly used for data-parallel communication will be pipelined.
This behavior can also be steered more granularly by enabling individual
flags.
* `xla_gpu_enable_pipelined_all_gather`
* `xla_gpu_enable_pipelined_all_reduce`
* `xla_gpu_enable_pipelined_reduce_scatter`
* Unrolling while loops by a factor of two. Breaks down the loop-barrier
potentially leading to a better compute-communication overlap and less
copies.
* `xla_gpu_enable_while_loop_double_buffering`
* Latency Hiding Scheduler will do most of the work to hide the communication
latency.
* `xla_gpu_enable_latency_hiding_scheduler`
* To maximize networking bandwidth, combiner passes will combine pipelined
collectives to the maximum available memory. The optimization does not kick
in if the loop is already unrolled in the input HLO.
* [all_gather_combiner](https://github.com/openxla/xla/blob/5b54d0e9cf34f4e5ab05b3752ecb390145ca5716/xla/service/gpu/transforms/collectives/all_gather_combiner.cc#L78)
* [all_reduce_combiner](https://github.com/openxla/xla/blob/5b54d0e9cf34f4e5ab05b3752ecb390145ca5716/xla/service/gpu/transforms/collectives/all_reduce_combiner.cc#L76)
* [reduce_scatter_combiner](https://github.com/openxla/xla/blob/5b54d0e9cf34f4e5ab05b3752ecb390145ca5716/xla/service/gpu/transforms/collectives/reduce_scatter_combiner.cc#L76)
## Memory Fitting Level
Another effort level option controls the degree to which the compiler will
attempt to make the resulting program "fit in memory", where "fit" and "memory"
have backend-dependent meanings (for example, in XLA:TPU, this option controls
the degree to which the compiler works to keep the TPU's high-bandwidth memory
(HBM) usage below the HBM capacity). It can be set via the
[memory_fitting_level](https://github.com/openxla/xla/blob/5b54d0e9cf34f4e5ab05b3752ecb390145ca5716/xla/pjrt/proto/compile_options.proto#L79)
field of the ExecutableBuildOptionsProto message, or the
[memory_fitting_level](https://github.com/openxla/xla/blob/f4d624b6811c28925c3006f5b779f1149b3b39ac/xla/xla.proto#L1588)
field of the ExecutionOptions message.
As with optimization level, the exact meaning of each effort level value is
backend-dependent, but the following table describes the expected effect as a
general guideline:
| Level | Use Case |
| :-------- | :---------------------------------------------------------------------- |
| EFFORT_O0 | Minimal effort to fit (fail compilation as quickly as possible instead) |
| EFFORT_O1 | Reduced effort to fit |
| EFFORT_O2 | Significant effort to fit (suitable default for production workloads) |
| EFFORT_O3 | Expensive or experimental algorithms to reduce memory usage |
third_party/xla/docs/emitters.md
Vendored
+539
View File
@@ -0,0 +1,539 @@
# XLA:GPU Emitters
There are three ways how to generate code for HLO in XLA:GPU.
![img](./images/emitters-codegen-types.jpg)
1. Replacing HLO with custom calls to external libraries, e.g. [NVidia cuBLAS](https://developer.nvidia.com/cublas), [cuDNN](https://developer.nvidia.com/cudnn).
2. Tiling HLO to block-level and then using [OpenAI Triton](https://openai.com/index/triton/).
3. Using XLA Emitters to progressively lower HLO to LLVM IR.
This document is focused on XLA:GPU Emitters.
## Hero-based codegen
There are 7 emitter types in XLA:GPU. Each emitter type corresponds to a "hero"
of the fusion, i.e. the most important op in the fused computation that shapes
the code generation for the whole fusion.
![img](./images/emitters-hero-types.jpg)
For example, the tranpose emitter will be selected if there is a
`HloTransposeInstruction` within the fusion that requires using shared memory to
improve the memory reading and writing patterns. The reduction emitter generates
reductions using shuffles and shared memory. The loop emitter is the default
emitter. If a fusion does not have a hero for which we have a special emitter,
then the loop emitter will be used.
## High-level overview
The code consists of the following big building blocks:
- Computation partitioner - splitting an HLO fusion computation into functions
- Emitters - converting partitioned HLO fusion to MLIR (`xla_gpu`, `tensor`, `arith`, `math`, `scf` dialects)
- Compilation pipeline - optimizes and lowers IR to LLVM
![img](./images/emitters-pipeline-overview.jpg)
## Partitioning
See [computation_partitioner.h](https://github.com/openxla/xla/blob/ca62f3e1bc9ea1d808c3a4de0a78bae7453389eb/xla/codegen/emitters/computation_partitioner.h).
Non-elementwise HLO instructions cannot always be emitted together. Consider the
following HLO graph:
```
param
|
log
| \
| transpose
| /
add
```
If we emit this in a single function, the `log` will be accessed at two
different indices for each element of the `add`. The old emitters solve this
problem by generating the `log` twice. For this particular graph, this is not
a problem, but when there are multiple splits, the code size grows
exponentially.
Here, we solve this problem by partitioning the graph into pieces that can be
safely emitted as one function. The criteria are:
- Instructions that have only one user are safe to emit together with their
user.
- Instructions that have multiple users are safe to emit together with their
users if they are accessed through the same indices by all users.
In the example above, the `add` and `tranpose` access different indices of the
`log`, so it is not safe to emit it together with them.
The graph is therefore partitioned into three functions (each containing just
one instruction).
The same is applicable to the following example with `slice` and `pad` of `add`.
![img](./images/emitters-partitioning.jpg)
## Elemental emission
See [elemental_hlo_to_mlir.h](https://github.com/openxla/xla/blob/ca62f3e1bc9ea1d808c3a4de0a78bae7453389eb/xla/codegen/emitters/elemental_hlo_to_mlir.h).
Elemental emission creates loops and math/arith ops for `HloInstructions`. For
the most part, this is straightforward, but there are some interesting things
going on here.
### Indexing transformations
Some instructions (`transpose`, `broadcast`, `reshape`, `slice`, `reverse` and
a few more) are purely transformations on indices: to produce an element of the
result, we need to produce some other element of the input. For this, we can
reuse XLA's [indexing_analysis](https://openxla.org/xla/indexing), which has
functions to produce the output to input mapping for an instruction.
For example, for a `transpose` from `[20,40]` to `[40,20]`, it will produce the
following indexing map (one symbolic expression per input dimension; d0 and d1
are the output dimensions):
```
(d0, d1) -> d1
(d0, d1) -> d0
```
So for these pure index transformation instructions, we can simply get the map,
apply it to the output indices, and produce the input at the resulting index.
Similarly, the `pad` op uses indexing maps and constraints for most of the
implementation. `pad` is also an indexing transformation with some added checks
to see if we return an element of the input or the padding value.
### Tuples
We do not support internal `tuple`s. We also do not support nested tuple
outputs. All XLA graphs that use these features can be converted to graphs that
do not.
### Gather
We only support canonical gathers as produced by [`gather_simplifier`](
https://github.com/openxla/xla/blob/main/xla/hlo/transforms/simplifiers/gather_simplifier.h).
## Subgraph functions
For a subgraph of a computation with parameters `%p0` to `%p_n`, and subgraph
roots with `r` dimensions and element types (`e0` to `e_m`), we use the
following MLIR function signature:
``````
(%p0: tensor<...>, %p1: tensor<...>, ..., %pn: tensor<...>,
%i0: index, %i1: index, ..., %i_r-1: index) -> (e0, ..., e_m)
``````
That is, we have one tensor input per computation parameter, one index input per
dimension of the output, and one result per output.
To emit a function, we simply use the elemental emitter above, and recursively
emit its operands until we reach the edge of the subgraph. Then, we:emit a
`tensor.extract` for parameters or emit a `func.call` for other subgraphs
## Entry function
Each emitter type differs in how it generates the entry function, i.e. the
function for the hero. The entry function is different from the functions above,
since it has no indices as inputs (just the thread and block IDs) and actually
needs to write the output somewhere. For the loop emitter, this is fairly
straightforward, but the transpose and reduction emitters have non-trivial write
logic.
The signature of the entry computation is:
```
(%p0: tensor<...>, ..., %pn: tensor<...>,
%r0: tensor<...>, ..., %rn: tensor<...>) -> (tensor<...>, ..., tensor<...>)
```
Where like before, the `%pn`s are the parameters of the computation, and the
`%rn`s are the results of the computation. The entry computation takes the
results as tensors, `tensor.insert`s updates into them, and then returns them.
No other uses of the output tensors are allowed.
## Compilation pipeline
### Loop emitter
See [loop.h](https://github.com/openxla/xla/blob/cfd16b7f21feff17635c782f4489c0f478178eb9/xla/backends/gpu/codegen/emitters/loop.h#L4).
Let's study the most important passes of the MLIR compilation pipeline using the
HLO for the GELU function.
![img](./images/emitters-gelu.png)
This HLO computation only has elementwise ops, constants and broadcasts. It will
be emitted using the loop emitter.
#### MLIR Conversion
After conversion to MLIR we get an `xla_gpu.loop` that depends on
`%thread_id_x` and `%block_id_x` and defines the loop that traverses all
elements of the output linearly to guarantee coalesced writes.
On every iteration of this loop we call
```
%pure_call = xla_gpu.pure_call @gelu(%input, %dim0, %dim1, %dim2)
: (tensor<6x512x4096xbf16>, index, index, index) -> bf16
```
to compute elements of the root operation. Note, that we have only one outlined
function for `@gelu`, because the partitioner did not detect a tensor that has 2
or more various access patterns.
```
#map = #xla_gpu.indexing_map<"(th_x, bl_x)[vector_index] -> ("
"bl_x floordiv 4096, (bl_x floordiv 8) mod 512, (bl_x mod 8) * 512 + th_x * 4 + vector_index),"
"domain: th_x in [0, 127], bl_x in [0, 24575], vector_index in [0, 3]">
func.func @main(%input: tensor<6x512x4096xbf16> , %output: tensor<6x512x4096xbf16>)
-> tensor<6x512x4096xbf16> {
%thread_id_x = gpu.thread_id x {xla.range = [0 : index, 127 : index]}
%block_id_x = gpu.block_id x {xla.range = [0 : index, 24575 : index]}
%xla_loop = xla_gpu.loop (%thread_id_x, %block_id_x)[%vector_index] -> (%dim0, %dim1, %dim2)
in #map iter_args(%iter = %output) -> (tensor<6x512x4096xbf16>) {
%pure_call = xla_gpu.pure_call @gelu(%input, %dim0, %dim1, %dim2)
: (tensor<6x512x4096xbf16>, index, index, index) -> bf16
%inserted = tensor.insert %pure_call into %iter[%dim0, %dim1, %dim2] : tensor<6x512x4096xbf16>
xla_gpu.yield %inserted : tensor<6x512x4096xbf16>
}
return %xla_loop : tensor<6x512x4096xbf16>
}
func.func private @gelu(%arg0: tensor<6x512x4096xbf16>, %i: index, %j: index, %k: index) -> bf16 {
%cst = arith.constant 5.000000e-01 : bf16
%cst_0 = arith.constant 1.000000e+00 : bf16
%cst_1 = arith.constant 7.968750e-01 : bf16
%cst_2 = arith.constant 4.467770e-02 : bf16
%extracted = tensor.extract %arg0[%i, %j, %k] : tensor<6x512x4096xbf16>
%0 = arith.mulf %extracted, %extracted : bf16
%1 = arith.mulf %0, %extracted : bf16
%2 = arith.mulf %1, %cst_2 : bf16
%3 = arith.addf %extracted, %2 : bf16
%4 = arith.mulf %3, %cst_1 : bf16
%5 = math.tanh %4 : bf16
%6 = arith.addf %5, %cst_0 : bf16
%7 = arith.mulf %6, %cst : bf16
%8 = arith.mulf %extracted, %7 : bf16
return %8 : bf16
}
```
#### Inliner
After `@gelu` is inlined, we get a single `@main` function. It can happen that
the same function is called twice or more. In this case we don't inline. More
details on the inlining rules can be found in
[xla_gpu_dialect.cc](https://github.com/openxla/xla/blob/39fc9c0c3e06b29e27f2aebdb21fca15754de928/xla/backends/gpu/codegen/emitters/ir/xla_gpu_dialect.cc).
```
func.func @main(%arg0: tensor<6x512x4096xbf16>, %arg1: tensor<6x512x4096xbf16>) -> tensor<6x512x4096xbf16> {
...
%thread_id_x = gpu.thread_id x {xla.range = [0 : index, 127 : index]}
%block_id_x = gpu.block_id x {xla.range = [0 : index, 24575 : index]}
%xla_loop = xla_gpu.loop (%thread_id_x, %block_id_x)[%vector_index] -> (%dim0, %dim1, %dim2)
in #map iter_args(%iter = %output) -> (tensor<6x512x4096xbf16>) {
%extracted = tensor.extract %input[%dim0, %dim1, %dim2] : tensor<6x512x4096xbf16>
%0 = arith.mulf %extracted, %extracted : bf16
%1 = arith.mulf %0, %extracted : bf16
%2 = arith.mulf %1, %cst : bf16
%3 = arith.addf %extracted, %2 : bf16
%4 = arith.mulf %3, %cst_0 : bf16
%5 = math.tanh %4 : bf16
%6 = arith.addf %5, %cst_1 : bf16
%7 = arith.mulf %6, %cst_2 : bf16
%8 = arith.mulf %extracted, %7 : bf16
%inserted = tensor.insert %8 into %iter[%dim0, %dim1, %dim2] : tensor<6x512x4096xbf16>
xla_gpu.yield %inserted : tensor<6x512x4096xbf16>
}
return %xla_loop : tensor<6x512x4096xbf16>
}
```
#### `xla_gpu` to `scf` conversion
See [lower_xla_gpu_to_scf.cc](https://github.com/openxla/xla/blob/39fc9c0c3e06b29e27f2aebdb21fca15754de928/xla/backends/gpu/codegen/emitters/transforms/lower_xla_gpu_to_scf.cc).
`xla_gpu.loop` represents a loop nest with a boundary check inside. If the loop
inductions variables are out of bounds of the indexing map domain, then this
iteration is skipped. It means, that the loop is converted to 1 or more nested
`scf.for` ops with an `scf.if` inside.
```
%xla_loop = scf.for %vector_index = %c0 to %c4 step %c1 iter_args(%iter = %output) -> (tensor<6x512x4096xbf16>) {
%2 = arith.cmpi sge, %thread_id_x, %c0 : index
%3 = arith.cmpi sle, %thread_id_x, %c127 : index
%4 = arith.andi %2, %3 : i1
%5 = arith.cmpi sge, %block_id_x, %c0 : index
%6 = arith.cmpi sle, %block_id_x, %c24575 : index
%7 = arith.andi %5, %6 : i1
%inbounds = arith.andi %4, %7 : i1
%9 = scf.if %inbounds -> (tensor<6x512x4096xbf16>) {
%dim0 = xla_gpu.apply_indexing #map(%thread_id_x, %block_id_x)[%vector_index]
%dim1 = xla_gpu.apply_indexing #map1(%thread_id_x, %block_id_x)[%vector_index]
%dim2 = xla_gpu.apply_indexing #map2(%thread_id_x, %block_id_x)[%vector_index]
%extracted = tensor.extract %input[%dim0, %dim1, %dim2] : tensor<6x512x4096xbf16>
// ... more arithmetic operations
%29 = arith.mulf %extracted, %28 : bf16
%inserted = tensor.insert %29 into %iter[%dim0, %dim1, %dim2] : tensor<6x512x4096xbf16>
scf.yield %inserted : tensor<6x512x4096xbf16>
} else {
scf.yield %iter : tensor<6x512x4096xbf16>
}
scf.yield %9 : tensor<6x512x4096xbf16>
}
```
#### Flatten tensors
See [flatten_tensors.cc](https://github.com/openxla/xla/blob/39fc9c0c3e06b29e27f2aebdb21fca15754de928/xla/backends/gpu/codegen/emitters/transforms/flatten_tensors.cc).
The N-d tensors are projected onto 1D. This will simplify the vectorization and
the lowering to LLVM because every tensor access now corresponds to how the data
is aligned in memory.
```
#map = #xla_gpu.indexing_map<"(th_x, bl_x, vector_index) -> (th_x * 4 + bl_x * 512 + vector_index),"
"domain: th_x in [0, 127], bl_x in [0, 24575], vector_index in [0, 3]">
func.func @main(%input: tensor<12582912xbf16>, %output: tensor<12582912xbf16>) -> tensor<12582912xbf16> {
%xla_loop = scf.for %vector_index = %c0 to %c4 step %c1 iter_args(%iter = %output) -> (tensor<12582912xbf16>) {
%dim = xla_gpu.apply_indexing #map(%thread_id_x, %block_id_x, %vector_index)
%extracted = tensor.extract %input[%dim] : tensor<12582912xbf16>
%2 = arith.mulf %extracted, %extracted : bf16
%3 = arith.mulf %2, %extracted : bf16
%4 = arith.mulf %3, %cst_2 : bf16
%5 = arith.addf %extracted, %4 : bf16
%6 = arith.mulf %5, %cst_1 : bf16
%7 = math.tanh %6 : bf16
%8 = arith.addf %7, %cst_0 : bf16
%9 = arith.mulf %8, %cst : bf16
%10 = arith.mulf %extracted, %9 : bf16
%inserted = tensor.insert %10 into %iter[%dim] : tensor<12582912xbf16>
scf.yield %inserted : tensor<12582912xbf16>
}
return %xla_loop : tensor<12582912xbf16>
}
```
#### Vectorization
See [vectorize_loads_stores.cc](https://github.com/openxla/xla/blob/39fc9c0c3e06b29e27f2aebdb21fca15754de928/xla/backends/gpu/codegen/emitters/transforms/vectorize_loads_stores.cc).
The pass analyses the indices in the `tensor.extract` and `tensor.insert` ops
and if they are produced by `xla_gpu.apply_indexing` that accesses the elements
contiguously w.r.t. to the `%vector_index` and the access is aligned, then
`tensor.extract` is converted to `vector.transfer_read` and hoisted out of the
loop.
In this particular case, there is an indexing map
`(th_x, bl_x, vector_index) -> (th_x * 4 + bl_x * 512 + vector_index)` used to
compute elements to extract and insert in a `scf.for` loop from 0 to 4.
Therefore, both `tensor.extract` and `tensor.insert` can be vectorized.
```
func.func @main(%input: tensor<12582912xbf16>, %output: tensor<12582912xbf16>) -> tensor<12582912xbf16> {
%vector_0 = arith.constant dense<0.000000e+00> : vector<4xbf16>
%0 = xla_gpu.apply_indexing #map(%thread_id_x, %block_id_x, %c0)
%2 = vector.transfer_read %input[%0], %cst {in_bounds = [true]} : tensor<12582912xbf16>, vector<4xbf16>
%xla_loop:2 = scf.for %vector_index = %c0 to %c4 step %c1
iter_args(%iter = %output, %iter_vector = %vector_0) -> (tensor<12582912xbf16>, vector<4xbf16>) {
%5 = vector.extract %2[%vector_index] : bf16 from vector<4xbf16>
%6 = arith.mulf %5, %5 : bf16
%7 = arith.mulf %6, %5 : bf16
%8 = arith.mulf %7, %cst_4 : bf16
%9 = arith.addf %5, %8 : bf16
%10 = arith.mulf %9, %cst_3 : bf16
%11 = math.tanh %10 : bf16
%12 = arith.addf %11, %cst_2 : bf16
%13 = arith.mulf %12, %cst_1 : bf16
%14 = arith.mulf %5, %13 : bf16
%15 = vector.insert %14, %iter_vector [%vector_index] : bf16 into vector<4xbf16>
scf.yield %iter, %15 : tensor<12582912xbf16>, vector<4xbf16>
}
%4 = vector.transfer_write %xla_loop#1, %output[%0] {in_bounds = [true]}
: vector<4xbf16>, tensor<12582912xbf16>
return %4 : tensor<12582912xbf16>
}
```
#### Loop unrolling
See [optimize_loops.cc](https://github.com/openxla/xla/blob/39fc9c0c3e06b29e27f2aebdb21fca15754de928/xla/backends/gpu/codegen/emitters/transforms/optimize_loops.cc).
The loop unrolling finds `scf.for` loops that can be unrolled. In this case, the
loop over the elements of the vector disappears.
```
func.func @main(%input: tensor<12582912xbf16>, %arg1: tensor<12582912xbf16>) -> tensor<12582912xbf16> {
%cst_0 = arith.constant dense<0.000000e+00> : vector<4xbf16>
%dim = xla_gpu.apply_indexing #map(%thread_id_x, %block_id_x, %c0)
%2 = vector.transfer_read %input[%dim], %cst {in_bounds = [true]} : tensor<12582912xbf16>, vector<4xbf16>
%3 = vector.extract %2[%c0] : bf16 from vector<4xbf16>
...
%13 = vector.insert %12, %cst_0 [%c0] : bf16 into vector<4xbf16>
%14 = vector.extract %2[%c1] : bf16 from vector<4xbf16>
...
%24 = vector.insert %23, %13 [%c1] : bf16 into vector<4xbf16>
%25 = vector.extract %2[%c2] : bf16 from vector<4xbf16>
...
%35 = vector.insert %34, %24 [%c2] : bf16 into vector<4xbf16>
%36 = vector.extract %2[%c3] : bf16 from vector<4xbf16>
...
%46 = vector.insert %45, %35 [%c3] : bf16 into vector<4xbf16>
%47 = vector.transfer_write %46, %arg1[%dim] {in_bounds = [true]} : vector<4xbf16>, tensor<12582912xbf16>
return %47 : tensor<12582912xbf16>
}
```
#### Conversion to LLVM
We mostly use the standard LLVM lowerings, but there are a few special passes.
We cannot use the `memref` lowerings for tensors, since we don't bufferize the
IR and our ABI is not compatible with the `memref` ABI. Instead, we have a
custom lowering directly from tensors to `LLVM`.
- The lowering of tensors is done in [lower_tensors.cc](https://github.com/openxla/xla/blob/39fc9c0c3e06b29e27f2aebdb21fca15754de928/xla/backends/gpu/codegen/emitters/transforms/lower_tensors.cc). `tensor.extract` is
lowered to `llvm.load`, `tensor.insert` to `llvm.store`, in the obvious way.
- [propagate_slice_indices](https://github.com/openxla/xla/blob/39fc9c0c3e06b29e27f2aebdb21fca15754de928/xla/backends/gpu/codegen/emitters/transforms/propagate_slice_indices.cc) and [merge_pointers_to_same_slice](https://github.com/openxla/xla/blob/39fc9c0c3e06b29e27f2aebdb21fca15754de928/xla/backends/gpu/codegen/emitters/transforms/merge_pointers_to_same_slice.cc) together
implement a detail of buffer assignment and XLA's ABI: if two tensors share
the same buffer slice, they are only passed once. These passes deduplicate the
function arguments.
```
llvm.func @__nv_tanhf(f32) -> f32
llvm.func @main(%arg0: !llvm.ptr, %arg1: !llvm.ptr) {
%11 = nvvm.read.ptx.sreg.tid.x : i32
%12 = nvvm.read.ptx.sreg.ctaid.x : i32
%13 = llvm.mul %11, %1 : i32
%14 = llvm.mul %12, %0 : i32
%15 = llvm.add %13, %14 : i32
%16 = llvm.getelementptr inbounds %arg0[%15] : (!llvm.ptr, i32) -> !llvm.ptr, bf16
%17 = llvm.load %16 invariant : !llvm.ptr -> vector<4xbf16>
%18 = llvm.extractelement %17[%2 : i32] : vector<4xbf16>
%19 = llvm.fmul %18, %18 : bf16
%20 = llvm.fmul %19, %18 : bf16
%21 = llvm.fmul %20, %4 : bf16
%22 = llvm.fadd %18, %21 : bf16
%23 = llvm.fmul %22, %5 : bf16
%24 = llvm.fpext %23 : bf16 to f32
%25 = llvm.call @__nv_tanhf(%24) : (f32) -> f32
%26 = llvm.fptrunc %25 : f32 to bf16
%27 = llvm.fadd %26, %6 : bf16
%28 = llvm.fmul %27, %7 : bf16
%29 = llvm.fmul %18, %28 : bf16
%30 = llvm.insertelement %29, %8[%2 : i32] : vector<4xbf16>
...
}
```
### Transpose emitter
Let's consider a slightly more involved example.
![img](./images/emitters-transpose.png)
The transpose emitter differs from the loop emitter only in how the entry
function is generated.
```
func.func @transpose(%arg0: tensor<20x160x170xf32>, %arg1: tensor<170x160x20xf32>) -> tensor<170x160x20xf32> {
%thread_id_x = gpu.thread_id x {xla.range = [0 : index, 127 : index]}
%block_id_x = gpu.block_id x {xla.range = [0 : index, 959 : index]}
%shmem = xla_gpu.allocate_shared : tensor<32x1x33xf32>
%xla_loop = xla_gpu.loop (%thread_id_x, %block_id_x)[%i, %j]
-> (%input_dim0, %input_dim1, %input_dim2, %shmem_dim0, %shmem_dim1, %shmem_dim2)
in #map iter_args(%iter = %shmem) -> (tensor<32x1x33xf32>) {
%extracted = tensor.extract %arg0[%input_dim0, %input_dim1, %input_dim2] : tensor<20x160x170xf32>
%0 = math.exp %extracted : f32
%inserted = tensor.insert %0 into %iter[%shmem_dim0, %shmem_dim1, %shmem_dim2] : tensor<32x1x33xf32>
xla_gpu.yield %inserted : tensor<32x1x33xf32>
}
%synced_tensor = xla_gpu.sync_threads %xla_loop : tensor<32x1x33xf32>
%xla_loop_0 = xla_gpu.loop (%thread_id_x %block_id_x)[%i, %j] -> (%dim0, %dim1, %dim2)
in #map1 iter_args(%iter = %arg1) -> (tensor<170x160x20xf32>) {
// indexing computations
%extracted = tensor.extract %synced_tensor[%0, %c0, %1] : tensor<32x1x33xf32>
%2 = math.absf %extracted : f32
%inserted = tensor.insert %2 into %iter[%3, %4, %1] : tensor<170x160x20xf32>
xla_gpu.yield %inserted : tensor<170x160x20xf32>
}
return %xla_loop_0 : tensor<170x160x20xf32>
}
```
In this case, we generate two `xla_gpu.loop` ops. The first one performs
coalesced reads from the input and writes the result to the shared memory.
The shared memory tensor is created using `xla_gpu.allocate_shared` op.
After the threads are synchronized using `xla_gpu.sync_threads`, the second
`xla_gpu.loop` reads the elements from the shared memory tensor and performs
coalesced writes to the output.
### Reproducer
In order to see the IR after every pass of the compilation pipeline, one can
launch `run_hlo_module` with the `--xla_dump_emitter_re=mlir-fusion` flag.
```
run_hlo_module --platform=CUDA --xla_disable_all_hlo_passes --reference_platform="" /tmp/gelu.hlo --xla_dump_emitter_re=mlir-fusion --xla_dump_to=<some_directory>
```
where `/tmp/gelu.hlo` contains
```
HloModule m:
gelu {
%param = bf16[6,512,4096] parameter(0)
%constant_0 = bf16[] constant(0.5)
%bcast_0 = bf16[6,512,4096] broadcast(bf16[] %constant_0), dimensions={}
%constant_1 = bf16[] constant(1)
%bcast_1 = bf16[6,512,4096] broadcast(bf16[] %constant_1), dimensions={}
%constant_2 = bf16[] constant(0.79785)
%bcast_2 = bf16[6,512,4096] broadcast(bf16[] %constant_2), dimensions={}
%constant_3 = bf16[] constant(0.044708)
%bcast_3 = bf16[6,512,4096] broadcast(bf16[] %constant_3), dimensions={}
%square = bf16[6,512,4096] multiply(bf16[6,512,4096] %param, bf16[6,512,4096] %param)
%cube = bf16[6,512,4096] multiply(bf16[6,512,4096] %square, bf16[6,512,4096] %param)
%multiply_3 = bf16[6,512,4096] multiply(bf16[6,512,4096] %cube, bf16[6,512,4096] %bcast_3)
%add_1 = bf16[6,512,4096] add(bf16[6,512,4096] %param, bf16[6,512,4096] %multiply_3)
%multiply_2 = bf16[6,512,4096] multiply(bf16[6,512,4096] %add_1, bf16[6,512,4096] %bcast_2)
%tanh_0 = bf16[6,512,4096] tanh(bf16[6,512,4096] %multiply_2)
%add_0 = bf16[6,512,4096] add(bf16[6,512,4096] %tanh_0, bf16[6,512,4096] %bcast_1)
%multiply_1 = bf16[6,512,4096] multiply(bf16[6,512,4096] %add_0, bf16[6,512,4096] %bcast_0)
ROOT %multiply_0 = bf16[6,512,4096] multiply(bf16[6,512,4096] %param, bf16[6,512,4096] %multiply_1)
}
ENTRY main {
%param = bf16[6,512,4096] parameter(0)
ROOT fusion = bf16[6,512,4096] fusion(%param), kind=kLoop, calls=gelu
}
```
## Links to code
* Compilation pipeline: [emitter_base.h](https://github.com/openxla/xla/blob/cfd16b7f21feff17635c782f4489c0f478178eb9/xla/backends/gpu/codegen/emitters/emitter_base.h)
* Optimization and conversion passes: [backends/gpu/codegen/emitters/transforms](https://github.com/openxla/xla/tree/39fc9c0c3e06b29e27f2aebdb21fca15754de928/xla/backends/gpu/codegen/emitters/transforms)
* Partition logic: [computation_partitioner.h](https://github.com/openxla/xla/blob/ca62f3e1bc9ea1d808c3a4de0a78bae7453389eb/xla/codegen/emitters/computation_partitioner.h)
* Hero-based emitters: [backends/gpu/codegen/emitters](https://github.com/openxla/xla/tree/cfd16b7f21feff17635c782f4489c0f478178eb9/xla/backends/gpu/codegen/emitters)
* XLA:GPU ops: [xla_gpu_ops.td](https://github.com/openxla/xla/blob/39fc9c0c3e06b29e27f2aebdb21fca15754de928/xla/backends/gpu/codegen/emitters/ir/xla_gpu_types.td)
* Correctness and lit tests: [backends/gpu/codegen/emitters/tests](https://github.com/openxla/xla/tree/30229c9836cafa6b05c6d42f0d918e5f8dc0b2dd/xla/backends/gpu/codegen/emitters/tests)
third_party/xla/docs/error_codes.md
Vendored
+21
View File
@@ -0,0 +1,21 @@
# XLA Error codes
This page is a list of all error codes emitted by the XLA compiler.
- [E0100](./errors/error_0100.md) - Runtime: Buffer Allocation Failure
- [E0101](./errors/error_0101.md) - Runtime: Program Allocation Failure
- [E0102](./errors/error_0102.md) - Runtime: Program Input Buffer Mismatch
- [E0200](./errors/error_0200.md) - Runtime: Core Halted Unexpectedly
- [E1000](./errors/error_1000.md) - Compile Time: HBM OOM
- [E1001](./errors/error_1001.md) - Compile Time: Scoped Vmem OOM
- [E1200](./errors/error_1200.md) - Compile Time: Host Offload Output Mismatch
- [E2001](./errors/error_2001.md) - Compile Time: Unsupported RHS DataType on
Hardware
- [E2002](./errors/error_2002.md) - Compile Time: Mosaic Input/Output
Misaligned Block and Tiling
- [E2003](./errors/error_2003.md) - Compile Time: Mosaic Unproven Memory
Access Alignment
- [E3000](./errors/error_3000.md) - Compile Time: SparseCore Allocation
Failure
- [E3001](./errors/error_3001.md) - Compile Time: SparseCore No Viable Logical
Replica Count
third_party/xla/docs/errors/error_0100.md
Vendored
+85
View File
@@ -0,0 +1,85 @@
# Error code: E0100
**Category:** Runtime: Buffer Allocation Failure
This error indicates that XLA:TPU runtime’s memory allocator failed to find a
suitable block of memory on the accelerator’s HBM for the requested allocation.
**Sample error message:**
```
ValueError: RESOURCE_EXHAUSTED: Error allocating device buffer: Attempting to allocate 8.00M. That was not possible. There are 6.43M free.; (0x0x1_HBM0)
```
**XLA backends:** TPU
## Overview
This error is thrown on:
- Failures of user-initiated buffer allocation via
[`jax.device_put`](https://docs.jax.dev/en/latest/_autosummary/jax.device_put.html)
or
- Failures of user-scheduled program's output allocations.
These failures are typically caused due to a couple of reasons:
- **Out of Memory (OOM):** The user is trying to allocate a chunk of memory
that is larger than the total amount of free memory available on the TPU’s
HBM.
- **Memory fragmentation:** The allocation fails because **no single
contiguous free block** in the memory space is large enough to satisfy the
requested size. The total amount of free memory is sufficient for the
allocation, but it is scattered across the memory space in small,
non-contiguous blocks.
The TPU runtime has a number of mechanisms in-place to retry allocation failures
including:
- If there are queued deallocations, the runtime retries failed allocations.
- On OOMs caused by a fragmentation, the runtime can automatically trigger a
defragmentation and a retry.
- The TPU runtime prioritizes buffer allocations over keeping programs loaded.
If a buffer allocation fails due to insufficient HBM, the system will evict
loaded TPU programs until enough memory is available for the buffer.
So an error encountered after the above mitigations typically require user
action.
## Debugging
- **Reduce your model's memory footprint**
- **Decrease batch size:** Reducing the batch size directly lowers memory
usage.
- **Parameter sharding:** For very large models, use techniques like model
parallelism or sharding to distribute parameters across the HBM of
multiple TPU cores or hosts.
- **Shorten sequence/context length:** For models that operate on
sequences (like language models), reducing the input sequence length can
significantly decrease the memory footprint.
- **Buffer donation:** Utilize framework features (such as: `jax.jit(...,
donate_argnums=...)`) to signal to XLA that certain input buffers can be
overwritten and reused for outputs. Read
[Buffer donation](https://docs.jax.dev/en/latest/buffer_donation.html)
for more details.
- **Optimize checkpoint strategy:** Instead of saving the entire model
state at once, consider saving only the model weights or using a sharded
checkpointing strategy.
- **Address memory layout and padding**
- TPU memory is allocated in chunks, and padding can increase the actual
size of tensors.
- **Ensure no memory leaks**
- Ensure references to `jax.Array` objects are not being held longer than
intended. Holding on to `jax.Array` objects might prevent automatic
de-allocation even after program compilation is completed.
See also [Error code: E1000](https://openxla.org/xla/errors/error_1000) for
other strategies you can use to reduce the amount of memory each program uses.
### Tooling
Enable the `tpu_log_allocations_on_oom` flag for which the allocator will dump a
detailed report of all current allocations when an OOM occurs, which can be
invaluable for debugging.
third_party/xla/docs/errors/error_0101.md
Vendored
+75
View File
@@ -0,0 +1,75 @@
# Error code: E0101
**Category:** Runtime: Program Allocation Failure
This error indicates that the XLA runtime on a TPU device failed to load a
compiled XLA program executable into the TPU's HBM.
**Sample error message:**
```
XlaRuntimeError: RESOURCE_EXHAUSTED: Error loading program 'jit_embedding_pipeline_step_fn': Attempting to reserve 29.49G at the bottom of memory. That was not possible. There are 147.64M free, 0B reserved, and 147.64M reservable. Scope: unknown..: while running replica 0 and partition 34 of a replicated computation (other replicas may have failed as well).
```
**XLA backends:** TPU
## Overview
This error is typically caused by one of the following reasons:
- **Program size exceeds available HBM:** The compiled XLA program, including
its instructions, static data, and any embedded constants, is larger than
the total amount of free HBM currently available on the specific TPU core(s)
where the program is being loaded.
- **HBM fragmentation:** While the total free HBM on the device might be
sufficient in aggregate, it is not available in a single, contiguous block
large enough to fit the entire program.
It's important to understand how the TPU runtime prioritizes memory. Buffer
allocations are privileged over loaded programs. If a buffer allocation fails,
the runtime will evict already loaded programs from HBM to free up space. This
can lead to a situation where a program that loaded successfully before now
fails with an OOM error, because the HBM is now occupied with more data buffers.
## Debugging
- **Reduce buffer memory footprint:** Freeing up memory used by data buffers
will leave more room for the program itself.
- **Decrease batch size:** This is one of the most effective ways to
reduce the amount of memory used for activations.
- **Parameter sharding:** For very large models, use model parallelism or
sharding techniques (like FSDP or Megascale) to distribute the model's
parameters and computation across multiple TPU cores or hosts. Note that
this can increase the network communication overhead due to splitting
tensors across multiple chips.
- **Shorten sequence/context length:** For models processing sequential
data (e.g., NLP models), reducing the sequence length can significantly
decrease memory usage.
- **Buffer donation:** Use framework features (e.g., `jax.jit(...,
donate_argnums=...)`) to allow XLA to reuse the memory of input buffers
for storing output, reducing peak memory usage.
- **Reduce program’s memory requirements for temporaries**
- Reduce programs memory usage for temporaries by using the
`tpu_shared_memory_percent` flag. Note that this might negatively affect
performance.
- **Optimize execution strategy/reduce serving load**
- **Manage program loading:** If you are JIT-compiling multiple functions,
be aware that each function can result in a program being loaded. Try to
structure your workload to minimize the number of concurrently loaded
programs.
- **Ensure no memory leaks**
- Ensure references to `jax.Array` objects are not being held longer than
intended. Holding on to `jax.Array` objects might prevent automatic
de-allocation even after program compilation is completed.
See also [Error code: E1000](https://openxla.org/xla/errors/error_1000) for
other strategies you can use to reduce the amount of memory each program uses.
### Tooling
- Enable the `tpu_log_allocations_on_oom` flag for which the allocator will
dump a detailed report of all current allocations when an OOM occurs, which
can be invaluable for debugging.
- Profile Your Program: Use the JAX memory profiler or the TensorFlow profiler
to get a detailed view of your program's memory usage over time. This can
help identify unexpected peaks in memory consumption.
third_party/xla/docs/errors/error_0102.md
Vendored
+58
View File
@@ -0,0 +1,58 @@
# Error code: E0102
**Category:** Runtime: Program Input Mismatch
This error occurs when the XLA runtime detects that an input buffer provided at
execution time does not match the metadata expected by a compiled program, such
as the buffer size or PJRT memory space kind.
**Sample error message:**
```
XlaRuntimeError: INVALID_ARGUMENT: Executable(jit_embedding_pipeline_step_fn) expected parameter 2482 of size 5242880 (bf16[16,1280,40]{2,1,0:T(8,128)(2,1)}) but got buffer with incompatible size 1638400 (bf16[16,1280,40]{1,2,0:T(8,128)(2,1)}): while running replica 0 and partition 0 of a replicated computation (other replicas may have failed as well).
```
**XLA backends:** TPU
## Overview
For buffer-size mismatches, the error message indicates both the expected and
actual sizes, as well as the tensor shapes and layouts. Note that these errors
might occur even if two tensors have the same shape but their size in memory can
be different if their physical layout (how the data is tiled and arranged on the
hardware) is different. For memory-space mismatches, the error message reports
that the parameter buffer is in an unexpected memory space.
These errors are predominantly caused by:
- **Checkpoint and XLA configuration mismatch:** A model is trained and a
checkpoint is saved. The physical layout of the weights in that checkpoint
is determined by the exact XLA version and configuration (e.g. XLA flags) at
that time. Later, this checkpoint is loaded in a different environment where
the configuration has changed. A new flag, a different default value, or a
change in the model/XLA code can cause the runtime to expect a different
physical layout for the weights. When the old buffer from the checkpoint is
passed to the new compiled XLA program, the runtime throws an error.
- **Hardware/topology-specific layouts:** The XLA compiler is free to choose
different physical layouts for tensors to optimize performance on different
hardware. A layout that is optimal for v4 TPU might be different from a v5
TPU, or even for different pod slices of the same chip (e.g., 4x4x4 vs 4x8).
The error occurs when a model is compiled with an assumption about one
topology's layout, but at runtime it is scheduled on a different topology,
or there is a bug in the compiler's layout logic for a specific piece of
hardware.
## Debugging
- **Ensure configuration consistency between model export and re-runs from
checkpoints**
- Avoid using old checkpoints with new code unless you are certain that no
layout-affecting changes have been made.
- If you suspect a checkpoint/configuration mismatch, the most reliable
solution is to re-export the saved model using the exact same (and
current) codebase and configuration that you are using for inference or
fine-tuning.
- Check for configuration changes (e.g. XLA flags) between the two runs.
- **Hardware/topology-specific layouts**
- Check for hardware version and topology mismatches if switching hardware
or topologies.
third_party/xla/docs/errors/error_0200.md
Vendored
+158
View File
@@ -0,0 +1,158 @@
# Error Code: E0200
**Category:** Runtime: Core Halted Unexpectedly
This error indicates that a TPU core stopped executing
instructions prematurely. This is a fatal error state where the hardware forces
a halt due to an unrecoverable fault, a violation of hardware constraints, or a
deliberate interrupt triggered by compiler-generated runtime assertions.
**Sample error message:**
```
INTERNAL: Accelerator device halted prematurely, perhaps due to an on-device check-failure. Node 0 halted unexpectedly at tag:pc TensorCoreSequencer:1:0x1d9 ...
```
**XLA backends:** TPU
## Overview
XLA compiles JAX programs into a sequence of low-level assembly instructions.
At runtime, the TPU device executes these instructions sequentially. A
"Core Halted Unexpectedly" error occurs when the TPU hardware encounters an
unrecoverable condition that prevents further execution, forcing the core into a
fatal "HALTED" state.
Because this error can stem from physical hardware failures, compiler bugs, or
user code issues (particularly in custom kernels), you must carefully analyze
the log messages to identify the specific cause.
## Debugging
To resolve this error, you must first identify which of the three specific
scenarios caused the unexpected halt. Check your logs for the specific text
signatures described below.
- **"observed errors are: [Hardware/Network/Power]"**: This indicates a
physical infrastructure failure. → Jump to
[Scenario 1: Infrastructure failures (hardware/newtwork/power)](#scenario_1_infrastructure_failures_hardwarenetworkpower)
- **"observed errors are: [User]"**: This indicates a hardware constraint
violation. → Jump to
[Scenario 2: Hardware constraint violations](#scenario_2_hardware_constraint_violations)
- The error message contains specific details such as the following keywords:
`BoundsCheck`, `scheckne`, `scheckeq`, `schecklt`, `scheckge`,
`scheckbetween`
This indicates that a compiler-generated assertion in the compiled program
failed during execution. → Jump to
[Scenario 3: XLA compiler-generated assertion failures](#scenario_3_xla_compiler-generated_assertion_failures)
### Scenario 1: Infrastructure failures (hardware/network/power)
**Signature:** The logs explicitly state `observed errors are: [Hardware]` or
`observed errors are: [Network]` or `observed errors are: [Power]`.
This indicates a physical infrastructure failure unrelated to your software or
model logic. The TPU chip, the network fabric connecting the chips, or the
power supply has failed.
- **Retry the job:** If the issue was a transient voltage dip or network flap,
a simple retry may work.
- **Identify and remove bad nodes:** If the error persists on the same
specific task or host, the hardware is likely defective. Use your cluster
management tools to "drain"/"cordon" the affected node and restart your job
on healthy nodes.
### Scenario 2: Hardware constraint violations
**Signature:** The logs state `observed errors are: [User]`.
This indicates that the XLA compiler generated an instruction that violated a
inviolable hardware constraint (e.g., an instruction attempting to access an
out-of-bounds memory address on HBM or Scratchpad memory). While labeled "User",
this is rarely caused by high-level user code.
- **File an XLA bug:** This is likely a compiler bug, the compiler should
never emit instructions that violate hardware specs.
[Please file a bug report](https://github.com/openxla/xla).
### Scenario 3: XLA compiler-generated assertion failures
**Signature:** The error message contains specific details on the
compiler-generated assertion that is failing. Look for the for the following
keywords:
- `BoundsCheck`, `scheckne`, `scheckeq`, `schecklt`, `scheckge`,
`scheckbetween`
This indicates that a compiler-generated assertion in the compiled program
failed during execution. Analyze the specific error message to determine the
sub-type.
#### Scenario 3.A: Launch group mismatch
**Sample error message:**
```
Core halted unexpectedly: INTERNAL: Accelerator device halted prematurely, perhaps due to an on-device check-failure. Node 0 halted unexpectedly at tag:pc TensorCoreSequencer:1:0x1d9 (from TensorCoreSequencer:1:0x309): scheckne: An unexpected leader shows up in the launch group with a different launch id than the current group leader.
```
**Cause:** This error typically occurs in multi-host TPU environments. It
indicates that the TPU cores, which are expected to execute the same program in
a synchronized manner (as part of a "launch group") have become out of sync.
Specifically, a TPU core joined a synchronization group with a different program
identifier than the current group leader, suggesting inconsistent programs
across hosts.
- **Verify XLA flags:** Ensure all hosts use the exact same `XLA_FLAGS`.
- **Check for consistent JAX programs:** Check that all hosts are executing
identical JAX programs. Verify docker images, libtpu versions, etc.
#### Scenario 3.B: Bounds check failure
**Sample error message:**
```
Core halted unexpectedly: INTERNAL: Accelerator device halted prematurely, perhaps due to an on-device check-failure. Node 0 halted unexpectedly at tag:pc TensorCoreSequencer:23:0x292 (from TensorCoreSequencer:23:0xd74a): BoundsCheck 92 [deref of %s931] for %937 = dma.hbm_to_vmem [thread:$0] /*hbm=*/%s931, /*size_in_granules=*/16384, /*vmem=*/%s935, /*dst_syncflagno=*/%s860, /*src_stride=*/512, /*dst_stride=*/128, /*steps_per_stride=*/8
```
**Cause:** The program tried to access memory outside of allocated bounds. The
error message often includes details about the memory access type (e.g.,
`dma.hbm_to_vmem`) and the address calculation.
- **Debug custom kernels:** If using Pallas, check your index calculations.
Use
[`pl.debug_print`](https://docs.jax.dev/en/latest/_autosummary/jax.experimental.pallas.debug_print.html)
or
[`checkify`](https://docs.jax.dev/en/latest/_autosummary/jax.experimental.checkify.check.html)
to validate tensor indices.
- **Check sharding:** Ensure sharding annotations are consistent with tensor
shapes.
#### Scenario 3.C: Mosaic/Pallas synchronization
**Sample error message:**
```
Core halted unexpectedly: INTERNAL: Accelerator device halted prematurely, perhaps due to an on-device check-failure. Node 0 halted unexpectedly at tag:pc TensorCoreSequencer:21:0xae5 (from TensorCoreSequencer:21:0x54c5): Semaphore (scratch argument 1) has a nonzero value upon exit from a Mosaic kernel. Make sure every DMA is awaited, and every semaphore signal is paired with a wait.
```
**Cause:**
This error is specific to code generated by the Mosaic compiler (used by Pallas
JAX). It indicates a synchronization issue within a custom kernel. TPUs use
semaphores to manage dependencies (e.g., ensuring a DMA is complete before
use). This error suggests a signal on a semaphore was not properly waited upon.
- **Audit synchronization:** Ensure every `dma_start` has a corresponding
`dma_wait`.
- **Check semaphores:** Verify that semaphore signals and waits are strictly
paired.
### Uncategorized issues
If your error log does not match Scenario 1, 2, or 3 (i.e., no "observed
errors", no "scheck" tags, and no specific bounds/semaphore messages):
- **Action:** This is likely an internal XLA bug.
[Please file a bug report](https://github.com/openxla/xla).
third_party/xla/docs/errors/error_1000.md
Vendored
+279
View File
@@ -0,0 +1,279 @@
# Error code: E1000
**Category:** Compile Time: HBM OOM
This error indicates that the program requires more High Bandwidth Memory (HBM)
than is physically available on the TPU device.
**Sample error messages:**
```
RESOURCE_EXHAUSTED: TPU TensorCore Hbm usage: 34.82G, SparseCore Hbm usage 174.10G, exceeding available bytes: 95.74G
```
```
RESOURCE_EXHAUSTED: XLA:TPU compile permanent error. Ran out of memory in memory space hbm. Used 49.34G of 32.00G hbm. Exceeded hbm capacity by 17.34G.
```
**XLA backends:** TPU
## Overview
XLA performs checks to ensure that the aggregate size of all necessary static
allocations fit in the device's HBM.
The compiler manages the TPU's fixed HBM capacity for several types of
allocations:
- **Program inputs and outputs:** Training batches, optimizer states etc.
- **TPU temporaries:** Dynamic memory required for intermediate calculations
(e.g. activations, gradients, etc).
- **Compiled binary:** The machine code for both TensorCore (TC) and
SparseCore (SC).
- **System overhead:** Reserved space for the XLA Runtime (e.g. infeed buffers
on older TPU generations).
- **Constants:** Constant values embedded in the HLO IR are allocated on HBM.
- **Compiler internals:** Program level and per-HLO allocations (e.g. routing
info for nodes in the mesh).
This error occurs when the XLA compiler cannot fit all of the above allocations
into the device HBM.
## Debugging
Carefully analyze the error message and logs to determine which category of HBM
OOM below best describes your error:
- **"TC Hbm usage: X, SC Hbm usage Y":** If the error explicitly breaks down
HBM usage, the aggregate TensorCore (TC) + SparseCore (SC) usage exceeds the
HBM limit. → Jump to
[Scenario 1. Balance TC and SC HBM usage](#scenario_1_balance_tc_and_sc_hbm_usage).
- **"Ran out of memory in memory space HBM"**: Check the logs for an
enumeration of the largest allocations on HBM.
- In case one or more unexpectedly large tensors (e.g. > 50% of HBM limit)
are present → Jump to
[Scenario 2. Out of memory due to unexpectedly large allocations](#scenario_2_out_of_memory_due_to_unexpectedly_large_allocations).
- If no unexpectedly large tensors are present in the logs → Jump to
[Scenario 3. Out of memory due to aggregate allocations](#scenario_3_out_of_memory_due_to_aggregate_allocations).
---
### Scenario 1. Balance TC and SC HBM usage
If the error explicitly breaks down usage, e.g., *"TC Hbm usage: X, SC Hbm usage
Y"*, this means the aggregate TensorCore (TC) + SparseCore (SC) usage exceeds
the HBM limit. Compare the two values to identify the bottleneck:
- **High SparseCore usage**
- **Optimize HBM stack usage:** HBM stack memory consumption scales with
`feature_width`, `max_unique_nz_per_row` and `logical_replica_count`.
You can reduce peak stack usage by tuning the
`--xla_sc_num_serialized_tables_to_optimize_hbm` flag which serializes
the processing of tables. This comes at the cost of reduced parallelism.
- **Check padding overhead:** SparseCore aligns embedding tables to 32B (8
floats). Tables with small feature widths (e.g., < 8 floats) incur
significant padding overhead, wasting HBM.
- **Reduce heap usage:** High values for `maximum_parallel_iterations`
increase the amount of input data prefetched into the HBM heap. Lowering
this value can free up significant memory.
- **Verify sharding:** Ensure embedding tables are properly mod-sharded
across all chips. See
[How limits translate to tables](https://openxla.org/xla/sparsecore).
- Check out
[SC: Performance and memory bottlenecks](https://openxla.org/xla/sparsecore#10_performance_memory_bottlenecks)
for more ideas.
- **High TensorCore usage**
- Proceed to
[Scenario 2](#scenario_2_out_of_memory_due_to_unexpectedly_large_allocations).
- **Balanced**
- If neither is individually excessive but the sum is too high, you are at
the chip's capacity. You must try lowering usage of both components.
Follow recommendations in all three sections.
### Scenario 2. Out of memory due to unexpectedly large allocations
If you observe the error message *"Ran out of memory in memory space HBM"* and
one or more unexpectedly large allocations are present in the logs (> 50% of HBM
limit), it is almost never a hardware capacity issue. It is typically a
configuration error. Check the XLA label (if present) of the large allocations
for hints on their JAX source code.
- **Remove debugging artifacts**
- Using
[jax.debug.print()](https://docs.jax.dev/en/latest/_autosummary/jax.debug.print.html)
in large-scale runs can force the compiler to materialize the full
tensor in HBM to transfer it to the CPU, breaking fusion and increasing
peak memory usage. Remove any left-over `jax.debug.print()`s.
- **Fix inefficient mesh shapes or sharding**
- Incorrect mesh shapes or missing sharding annotations can cause the
compiler to default to **replication** - forcing the compiler to try to
fit really large tensors on a single chip.
- Check the shapes of the large allocations and verify sharding is
correctly specified and propagated by XLA.
### Scenario 3. Out of memory due to aggregate allocations
If you observe the error message *"Ran out of memory in memory space HBM"* and
no unexpectedly large tensors are present in the logs, the program runs out of
capacity due to the aggregate sum of allocations exceeding the HBM limit. In
this case, it is often helpful to visualize the memory profile to identify the
specific buffers contributing to the peak usage. See
[Debug OOM errors with XProf](https://openxla.org/xla/oom_debugging) for a
step-by-step guide on identifying peak memory contributors.
Once you have identified some of the top contributors, use the following steps
to optimize the memory footprint.
#### Scenario 3.A Adjust configuration
You can often resolve OOMs with these configuration adjustments:
- **Reduce batch size:** The memory needed for intermediate activations and
gradients is directly proportional to the batch size. Reducing the batch
size can often help reduce memory usage, although you may need to retune
your learning rate, momentum, or optimizer hyperparameters to maintain model
stability.
- **Donate input buffers:** When JAX executes a computation it uses buffers on
the device for all inputs and outputs. If you know that one of the inputs is
not needed after the computation, and if it matches the shape and element
type of one of the outputs, you can specify that you want the corresponding
input buffer to be donated to hold an output. This will reduce the memory
required for the execution by the size of the donated buffer. You can
achieve this by specifying
[donate_argnums](https://docs.jax.dev/en/latest/buffer_donation.html)
parameter as an argument when using `jax.jit`.
- **Enable mixed precision (bfloat16):** Use bfloat16 or quantization (int8
etc) for the largest tensors in the program if the model architecture and
quality requirements allow. Note that this change can affect model behaviour
and should be considered carefully.
##### Micro-batching (optional)
If reducing the global batch size or increasing the chip count is not viable,
and the batch size per chip is not already minimized, you can try a
micro-batching strategy:
- Split each batch into `n` micro-batches;
- For each micro-batch, process the forward and backward pass;
- Once this is done, accumulate the gradients and update the weight as a
whole.
This process reduces the activation memory as we divided each batch into `n`
micro-batches, so that the if the original batch had size `M`, the activation
memory size becomes `M/n`.
**Potential issues:** - This process increases step time as we have multiple
forward and backward passes. - If the sizes of the model and micro-batch are too
different, you may face convergence issues in your model.
#### Scenario 3.B Optimize architecture and sharding
If configuration changes are insufficient, the model topology might be too large
for the current hardware setup.
- **Use newer TPU generations:** Newer TPUs generally offer more HBM per chip;
switch to newer TPU generations if available.
- **Run on a larger chip topology:** If the model weights are too large for
the existing topology, you can try sharding them across more chips.
- **Implement advanced sharding techniques:**
- Explore more advanced data, tensor, or pipeline parallelism approaches.
- Specify
[sharding hints](https://docs.jax.dev/en/latest/notebooks/Distributed_arrays_and_automatic_parallelization.html#constraining-shardings-of-intermediates-in-jitted-code)
for intermediate values and outputs.
Note that this may cause an increase in network communication overhead due
to splitting tensors across multiple chips.
- **Use JAX host offloading:** Host offloading techniques allow the user to
offload large tensors to the host CPU memory (e.g.
[activation offloading](https://docs.jax.dev/en/latest/notebooks/host-offloading.html#activation-offloading)
and
[optimizer state offloading](https://docs.jax.dev/en/latest/notebooks/host-offloading.html#optimizer-state-offloading)).
Note that host offloading techniques can severely impact performance, since
these operations will force the system to constantly move large tensors back
and forth between TPU HBM and CPU RAM.
#### Scenario 3.C Check tensor padding and alignment
Inefficient tensor shapes are a common, silent cause of OOMs on TPUs. To get
peak performance on TPU's, XLA pads tensor dimensions—typically to multiples of
128 for the minor-most dimension and 8 for the second-minor. This padding
affects both input arrays and intermediate tensors (HLO temporaries),
potentially inflating memory usage significantly, especially with small
dimension sizes. See
[Array Layouts](https://docs.jax.dev/en/latest/pallas/tpu/details.html#array-layouts).
- **Audit shapes of large buffers:** (On TPU v5 with default layouts)
- Hovering over a buffer in
[Xprof Memory Viewer](https://openxla.org/xprof/memory_viewer#memory_viewer_components)
brings up the buffer details card which contains buffer details
including padding information.
- *Example*: A shape of `(129, 1024)` might be padded to `(256, 1024)`,
resulting in nearly 50% memory waste.
- *Correction:* A shape of `(128, 1024)` requires no padding and incurs 0%
memory waste.
- **Align dimensions:** Ensure all large tensor dimensions (batch size,
embedding dimension, hidden size) are multiples of 128. Note that this
change can affect model behaviour and should be considered carefully.
#### Scenario 3.D Tune key memory impacting XLA flags
[Key memory flags](https://openxla.org/xla/flags_guidance#memory_flags) can be
tuned to trade-off performance for lower memory usage. However, this strategy
should be used as a last resort measure since it can adversely affect
performance.
#### Scenario 3.E Tune XLA rematerialization pass/manual checkpointing
If the model is close to fitting into memory, you can use the
[jax.checkpoint](https://docs.jax.dev/en/latest/_autosummary/jax.checkpoint.html)
decorator with `jax.grad` to manually control which intermediates are saved on
the forward pass versus recomputed on the backward pass. Note that this
operation may impact performance, since you are explicitly trading compute
cycles for HBM. Check out the JAX documentation for more information: -
[Gradient checkpointing with `jax.checkpoint` (`jax.remat`)](https://docs.jax.dev/en/latest/gradient-checkpointing.html) -
[Control autodiff’s saved values with `jax.checkpoint` (aka `jax.remat`)](https://docs.jax.dev/en/latest/notebooks/autodiff_remat.html) -
[JAX Memories and Host Offloading](https://docs.jax.dev/en/latest/notebooks/host-offloading.html)
Alternatively, you can force the `XLA::Rematerialization` pass to prioritize
memory savings, potentially at the cost of slower compilations:
| Flag | Description | Impact / Trade-off |
| --- | --- | --- |
| `--xla_tpu_max_hbm_size_mib` | Manually sets the limit on HBM size used by the Rematerialization pass. | Forces the compiler to work harder to fit the program into a limit smaller than the actual physical HBM. |
| `--xla_tpu_rematerialization_algo=PEAK_PRIORITY` | Focuses efforts at the points of peak memory usage. | Can be more efficient for aggressive memory reduction than the default algorithm. |
| `--xla_tpu_rematerialization_max_block_size_limit=32` | Controls the maximum number of instructions in a block that can be rematerialized at once. | Increasing this allows for memory savings at the cost of **significantly increases compile time**. |
| `--xla_tpu_rematerialization_block_effort_factor=10.0` | Defines the amount of effort (compile time) spent searching for blocks to rematerialize. | Higher values allow a more exhaustive search for memory savings at the cost of **increased compile times**. |
| `--xla_tpu_pre_fusion_remat=true` | Enables an additional Rematerialization pass *before* the fusion pass. | Can find more memory savings, but increases compile times and may **potentially impact numerical stability**. |
Note that making changes to XLA flags should be used as a last resort measure,
since it can adversely affect performance.
#### Scenario 3.F Use advanced profiling tools
[Debug OOM errors with XProf](https://openxla.org/xla/oom_debugging.md) provides
a tutorial on using the
[XProf Memory Viewer](https://openxla.org/xprof/memory_viewer) to visualize the
compiler's view of HBM usage.
This tool allows you to see peak memory allocation and buffer lifetimes, which
is crucial for understanding exactly what consumes HBM at the point of peak
utilization. For general profiling setup, see
[Getting started with Xprof](https://openxla.org/xprof#getting_started) and
[TensorBoard Profiling](https://docs.jax.dev/en/latest/profiling.html#xprof-tensorboard-profiling).
## Summary table
The following table contains a summary of potential interventions to solve OOM
errors and information that will help you decide what to do.
Intervention | Safe to do? (Will it change the behavior of the program?) | Potential gains | Telltale signs (is this actually the bottleneck that you're experiencing?)
----------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------
Using advanced sharding techniques | **Yes.** It almost never changes the numerical correctness of the experiment, although it can cause network communication overhead due to splitting tensors across multiple chips. | **Massive gains** (up to a 256x reduction) | Unexpectedly large individual allocations in the memory viewer (e.g., a single tensor replicated across all TPUs that is 256x bigger than the others). Active arrays showing as un-sharded in TensorBoard hooks.
Reducing batch size | **No.** It changes the training dynamics and usually requires retuning the learning rate. (Note: **Microbatching** is a safe alternative that reduces memory without changing behavior). | **Massive gains** (can save a factor of thousands). | "Temporaries" failing to allocate during gradient calculations. Seeing "JVP" in the operation name, and encountering many batch-size-shaped tensors in the memory profile.
Enabling mixed Precision (e.g., Bfloat16) | **Risky.** It alters numerical precision, which can change experiment results or cause the model to fail to converge entirely. | **Moderate gains** (typically a factor of 2x, as it halves memory usage). | The memory viewer confirms that the largest tensors are currently utilizing 32-bit floats (`float32`).
Manual checkpointing (`jax.checkpoint`) | **Yes.** It does not alter behavior; it merely trades computation time (flops) to save memory by recomputing tensors instead of storing them. | **Large gains** (e.g., can result in only half of the activations needing to exist in memory at the same time). | Multiple tensors of the exact same size filling up memory during a backward pass. Often accompanied by "JVP" in the operation name.
Donating input buffers (`donate_argnums`) | **Yes.** Maintains experiment integrity. If applied incorrectly, it will not corrupt data but will simply throw a clear error message. | **Marginal gains** (~1% memory savings). | There is no specific telltale sign, but it is considered a "free win" that is always worth trying.
Changing model dimensions | **No.** Directly alters the model's behavior. Modifying input or output dimensions can completely break compatibility with the dataset. | **Variable gains** depending on how drastically hidden dimensions or layers are reduced. | The Xprof memory viewer shows a large amount of memory wasted on "padding" because array dimensions are not powers of two or multiples of 128 (e.g., a dimension of 2050 instead of 2048).
Host offloading (CPU) | **Yes (numerically)**, but **No (performance)**. While mathematically safe, it is considered a "foot gun" that may cause severe speed bottlenecks due to data transfer between the CPU and TPU. | **Marginal gains** (the CPU only has about 3x the memory of the TPU). | Typically a last resort for massive optimizer states or for memory-heavy data preparation/pre-processing steps.
third_party/xla/docs/errors/error_1001.md
Vendored
+77
View File
@@ -0,0 +1,77 @@
# Error code: E1001
**Category:** Compile Time: Scoped Vmem OOM
This error indicates that the program requires more Scoped Vector Memory (Vmem)
than what was allocated.
**Sample Error Messages:**
```
RESOURCE_EXHAUSTED: Ran out of memory in memory space vmem while allocating on stack for %my-custom-kernel = bf16[2048,4096]{1,0:T(8,128)(2,1)} custom-call(...) ...
```
**XLA Backends:** TPU
## Overview
TPUs have Vector Memory (VMEM) which is a local scratchpad memory used
exclusively by the TensorCore (TC). The compiler manages Vmem for different
types of allocations:
* **Instruction-scoped allocations:** Temporary storage in Vmem while executing
a single HLO instruction. This includes operand span buffer (e.g. for double
buffering) and register spills.
* **Program-scoped allocations:** Allocations that live beyond the scope of
a single HLO instruction. These are usually HLO temporaries and intermediate
results that are inputs and/or outputs of HLO instructions.
A Compile Time Scoped Vmem OOM occurs when the instruction-scoped allocations
exceed the allocation limit for that instruction. This limit is controlled
- globally for the entire program via the flag
[--xla_tpu_scoped_vmem_limit_kib](https://openxla.org/xla/flags_guidance)
and
- per custom kernel via
[vmem_limit_bytes param](https://docs.jax.dev/en/latest/_autosummary/jax.experimental.pallas.tpu.CompilerParams.html#jax.experimental.pallas.tpu.CompilerParams.vmem_limit_bytes).
These errors are typically caused by an internal compiler bug or by a
custom kernel exceeding its allocation limit.
## Debugging
Carefully analyze the error message to identify if the error stems from a
custom kernel or a standard HLO. An error due to a custom kernel should have
the following signature:
```
Ran out of memory in memory space vmem while allocating on stack for %my-custom-call = <output-shape> custom-call(<params>), custom_call_target="tpu_custom_call" ...
```
- **Custom kernel scoped Vmem OOM**: If the error points to a custom kernel →
Jump to [Retune the Kernel](#retune_the_kernel).
- **Non-kernel Vmem issues**: If the Vmem OOM occurs due to a
non-custom-kernel op, it is likely an internal compiler bug.
[Please file a bug report on XLA](https://github.com/openxla/xla) with an
HLO dump.
### Retune the kernel
If the error originates from a custom kernel, use the following techniques to
reduce the kernel's memory requirement:
* **Adjust Block Sizes:** Reduce the block sizes (tile sizes) in your kernel
configuration, to lower Scoped Vmem usage.
* **Set Per-Kernel Scoped Vmem Limits:** Explicitly request the required
amount of memory for that specific kernel using the
[vmem_limit_bytes param](https://docs.jax.dev/en/latest/_autosummary/jax.experimental.pallas.tpu.CompilerParams.html#jax.experimental.pallas.tpu.CompilerParams.vmem_limit_bytes)
* **Modify Memory Coloring:** Explicitly color/constrain the kernel's
inputs/outputs to Vmem using
[pallas.tpu.with_memory_space_constraint](https://docs.jax.dev/en/latest/_autosummary/jax.experimental.pallas.tpu.with_memory_space_constraint.html).
Be careful not to color too many inputs outputs to Vmem, as that might cause
an overall Vmem OOM.
* **Adjust Vmem limit:** If kernel specific retuning is difficult or the issue
affects many kernels, you can adjust the global Vmem limit using the flag
[--xla_tpu_scoped_vmem_limit_kib](https://openxla.org/xla/flags_guidance).
third_party/xla/docs/errors/error_1200.md
Vendored
+60
View File
@@ -0,0 +1,60 @@
# Error code: E1200
**Category:** Compile Time: Host Offload Output Mismatch
This error occurs when a tensor explicitly offloaded to host memory is returned
as a program output, but the program's output signature is not configured to
expect host memory.
**Sample Error Messages:**
```
INVALID_ARGUMENT: Tensor which is moved to host (starting from tuple.64) is returned from the entry computation but the layout for this output is not set to host memory.
```
**XLA Backends:** TPU, GPU
## Overview
When the compiler encounters an annotation to offload a tensor to the host
(CPU), it tracks that tensor's location through the computation graph until one
of three events occurs:
1. **Move to Device:** A matching annotation moves the tensor back to the
accelerator.
2. **Host Computation:** The tensor is consumed by a host-side operation.
3. **Program End:** The tensor reaches the end of the program and becomes an
output.
This error is triggered in scenario **#3**. The tensor is physically located in
host memory at the end of the execution, but the XLA program's entry
computation signature defines that specific output as residing in **Device
Memory**. Because the compiler cannot implicitly change the entry computation's
interface, it raises an error.
## Debugging
To resolve this error, determine whether you intended for this tensor to be an
output on the Host or if it should have been moved back to the Device before
returning.
* **Intended to return on Host:** If you explicitly want this tensor to be
returned in host memory (avoiding a transfer back to device), you should
explicitly set the output memory space of the entry computation to **Host
Memory** for this specific output.
* **Intended to return on Device:** If the tensor was meant to stay on the
device or return to it before the program ends, you likely missed an
annotation. Insert a matching annotation to move the tensor back to the
device.
If the source of the offloaded tensor is unclear, or you cannot find where
the "move to device" annotation is missing, use XLA logging to trace the
instructions.
* **Enable logging:** If you are on Google Cloud TPU, rerun your program with
the following flag: `--vmodule=host_offloader=1`.
* **Analyze Logs:** Look for the "trace" output in the logs. This will show
the path of the tensor starting from the offload instruction. Use this to
pinpoint exactly where the tensor reaches the program boundary without being
moved back to the device.
third_party/xla/docs/errors/error_2001.md
Vendored
+88
View File
@@ -0,0 +1,88 @@
# Error code: E2001
**Category:** Compile Time: Unsupported RHS DataType on Hardware
This error occurs when the data type used for the **Right-Hand Side (RHS)**
operand in a matrix multiplication (e.g., `jax.lax.dot_general`, `jax.lax.conv`,
`jax.numpy.matmul`, or the `@` operator) is not natively supported by the
specific TPU generation being used.
**Sample error messages:**
```
INTERNAL: Mosaic failed to compile TPU kernel: Unsupported matmul RHS type on target: 'vector<256x256xi8>'
...
The MLIR operation involved:
%13440 = "tpu.matmul"(%13435, %13437, %13439) <dimension_numbers = #tpu.dot_dimension_numbers<...>
```
**XLA backends:** TPU
## Overview
The TPU's Matrix Multiply Unit (MXU) natively supports `Float32` operations on
all hardware generations.
However, native support for `BFloat16` and other quantized data types
(e.g., Int4, Int8, or Float8) varies by hardware generation. This error is
triggered when your kernel attempts to map a matrix multiplication to the MXU
using a data type that your specific TPU generation does not have the physical
circuitry to execute.
This error typically indicates that the compiler's **Canonicalization** pass—
which attempts to automatically convert unsupported types into supported ones
(e.g., via software emulation)—was unable to find a valid conversion rule or
was prevented from doing so because **Compatibility Mode** was disabled.
## Debugging
To resolve this error, you must align your data types with the capabilities of
your hardware. You have following options:
### 1. Cast to native types
The most reliable fix is to manually cast your operands to a hardware-supported
datatype (like `Float32` or `BFloat16` on TPU v4+) inside your kernel before the
matmul operation.
* **Why:** `Float32` is the universal data type supported natively by the
MXU on all TPU generations.
* **Trade-off:** This comes with a VPU (Vector Processing Unit) cost - the
cycles required to perform the cast, but it guarantees your kernel
will run on the current hardware.
### 2. Check Compatibility Mode
Typically the compiler can automatically handle these type mismatch issues in
**Compatibility Mode** which is enabled by default. Double check XLA configs
to make sure `--xla_mosaic_compat_mode` is not set to false.
This acts as a "polyfill," injecting software emulation sequences for operations
your hardware does not natively support.
**What Compatibility Mode enables:**
* **Mixed-precision MatMuls:** Allows mixing Integer operands with Float
accumulators by automatically inserting cast operations (e.g., extending
integers to `Float32` before the matmul).
* **Low-precision emulation:** On certain hardware generations, emulates
unsupported types like `4-bit` floating point (`4E2M1FN`) or `8-bit`
floating point (`8E4M3FN`) by extending them to supported types like
`BFloat16` or `Float32` before execution.
Note that this mode prioritizes compatibility over peak performance as emulation
requires additional instructions to convert data formats before the MXU can
operate on them.
### 3. Upgrade hardware or request support
If your algorithm strictly requires native performance for types like `Int4` or
`Float8` without the overhead of casting or emulation, you will need to run on
a newer TPU generation with native support.
**Feature request:** If you believe your hardware supports this operation, or if
the compiler is missing a valid emulation path even in Compatibility Mode,
please file a feature request. We usually guarantee that operations are forward
compatible. So if your kernel runs on a TPU generation then it should run on all
future generations, but it is not guaranteed to have emulation for older
generations (for some of which the casts would be very expensive).
third_party/xla/docs/errors/error_2002.md
Vendored
+53
View File
@@ -0,0 +1,53 @@
# Error code: E2002
**Category:** Compile Time: Mosaic Input/Output Misaligned Block and Tiling
This error occurs when the block shape of a kernel input or output does not
align with the default tiling of the datatype on the specific TPU hardware
being used.
**Sample error messages:**
```
UNIMPLEMENTED: Mosaic failed to compile TPU kernel: Failed to set window params
for input 0: Operand of shape (..., 256, 8192) has tiling (16, 128), but its
block shape (..., 8, 8192) is not divisible by tiling evenly nor matches the
full shape.
```
**XLA backends:** TPU
## Overview
Tensor cores (TC) in TPUs have two-dimensional vector registers. The two
dimensions are called **sublane** and **lane**. Since TPU compute units (e.g.,
MXU) operate at the granularity of vector registers, XLA arrays are laid out in
TPU memory in tiles.
This tiling (e.g., `8x128`) minimizes data transformations when feeding the
compute units. The exact tiling dimensions (sublanes × lanes) depend on the
hardware generation and the data type. For example, a common tiling for
most types is **8×128**.
At compile time, XLA enforces the following constraints for the minor and
2nd-minor dimensions of each kernel input/output:
1. **Divisibility:** The block dimension must be a multiple of the tile
dimension in the underlying tensor, or
2. **Full shape exception:** If the block dimension is not divisible, it must
be equal to the **full size** of that dimension in the underlying tensor.
This error is triggered when a block violates both conditions. For example,
loading a block of shape `(8, 100)` from a input of shape `(8, 1024)` on
hardware with shape `(8, 128)` tiling fails because `100` is not divisible by
`128` and `100 != 1024`. But, it would be allowed if the input shape was
`(32, 100)`.
## Debugging
To resolve this error, ensure your kernel's block shapes align with the
current hardware tiling. Modify your kernel code to align the block size such
that it is a multiple of the required tiling.
For example, if the error states the tiling is `(16, 128)` but your block shape
is `(8, 128)`, change the block spec such that the shape matches `(16, 128)`.
third_party/xla/docs/errors/error_2003.md
Vendored
+75
View File
@@ -0,0 +1,75 @@
# Error code: E2003
**Category:** Compile Time: Mosaic Unproven Memory Access Alignment
This error occurs when the compiler analyzes a memory access operation (such as
`vector.load`, `vector.store`, `tpu.load`, or `tpu.store`) and cannot
statically prove that the dynamic index used for a specific dimension is a
multiple of the required **tiling size**.
**Sample error messages:**
```
INTERNAL: Mosaic failed to compile TPU kernel: cannot statically prove that index in dimension 1 is a multiple of 128
at location: ...
The MLIR operation involved:
%14372 = "vector.load"(%14371, %93, %14363) : (memref<4x256xf32, #tpu.memory_space<vmem>>, index, index) -> vector<1x32xf32>
```
**XLA backends:** TPU
## Overview
When your kernel loads or stores a vector, the memory address
(calculated from the base pointer plus the dynamic index) must align with the
vector's **tiling size** on the hardware. For example, if a dimension is tiled
by 128 elements, the dynamic index used to access it must be `0`, `128`, `256`,
etc. Note that many operations (like vector loads and stores) have no such
requirements for static indices.
The compiler enforces this requirement using **static analysis**. It traces the
history of the index variable back through the arithmetic operations that
produced it (e.g., multiplications, additions). If the compiler cannot guarantee
(at compile time) that the resulting value will always be divisible by the
tiling size, it raises this error.
The compiler treats "proven misalignment" and "unknown alignment" identically.
So if you use an index that is mathematically guaranteed to be misaligned (e.g.,
`i * 128 + 32`), the compiler will raise the same error.
So this error can occur when
1. You use a runtime variable (dynamic index) to access memory.
2. The index calculation logic is too complex for the compiler to analyze.
3. The index is mathematically valid but lacks an explicit proof in the code.
4. Static analysis determines "proven misalignment".
## Debugging
To resolve this error you have the following options:
### 1. Assert alignment explicitly
If you know your index is valid but the compiler cannot prove it, use the
`tpu.assume_multiple` operation. This acts as a promise to the compiler that a
value is divisible by a specific factor.
### 2. Use aligned loads and rotate
In scenarios where the misalignment is intentional,
instead of loading a small, unaligned vector segment:
* Load a larger, fully aligned tile and then rotate the values by a dynamic
amount to shift the desired data into position (since vector slices with
dynamic start indices are not supported). or
* Reshape or pad the tensor so that the data starts at index 0 and the stride
between accesses matches the hardware alignment.
* Example: If you iterate over chunks of size 32 starting at offset 1,
your offsets are 1, 33, 65... (unaligned).
* Fix: Re-pack the data into a new tensor where the first chunk is at 0
and the dimension is padded to 128. Your offsets become 0, 128, 256...,
which satisfies the alignment requirement.
These methods consume more memory but often simplify kernel logic and eliminate
the need for manual alignment assertions.
third_party/xla/docs/errors/error_3000.md
Vendored
+89
View File
@@ -0,0 +1,89 @@
# Error code: E3000
**Category:** Compile Time: SparseCore Allocation Failure
This error occurs when the `XLA:SparseCore` compiler is unable to allocate a
contiguous block of memory in the specified memory space required by the current
SparseCore program.
**Sample error messages:**
```
INTERNAL:Failed to run pass pipeline. Hlo-Op: result.1:279:1: error: 'memref.alloca' op current allocation offset upper bound (140704 words) exceeds the legitimate user allocatable offset upper bound (131071 words) in memory space 201 when allocating 23440 words. result.1:279:1: note: see current operation: %232 = "memref.alloca"() <{operandSegmentSizes = array<i32: 0, 0>}> : () -> memref<23440xf32, 201>
```
**XLA backends:** TPU
## Overview
The SparseCore (SC) is a specialized processor for sparse workloads. It relies
on specific memory hierarchies to manage data movement efficiently. The XLA
compiler attempts to statically size and allocate buffers based on hardware
limits and user-defined shapes. This error indicates an **Out of Memory (OOM)**
condition during this allocation phase.
The error message typically specifies a memory space ID. Below are the common
memory spaces and their integer encodings:
| Memory Space ID | Name | Description |
| --- | --- | --- |
| **0** | **Smem** | Local Scalar Memory. Used for scalar registers and control flow. |
| **201** | **TileSpmem** | Tile-specific Scratchpad Memory. Fast, local SRAM available to a specific SC tile. |
| **202** | **Spmem** | Shared Scratchpad Memory. Used to opportunistically stage data (inputs, outputs, intermediates) to hide HBM latency. |
| **203** | **HBM** | High Bandwidth Memory. Large, shared memory used for embedding tables, heaps, and stacks. |
| **204** | **Sync Flags** | Synchronization primitives used for coordination. |
For a deep dive into SC and its memory hierarchy, refer to the
[SparseCore documentation](https://openxla.org/xla/sparsecore).
## Debugging
The resolution depends on which memory space failed the allocation.
### Scenario 1. HBM allocation failures (Memory Space ID: 203)
This error occurs if a single temporary allocation requested by the SparseCore
program is too large to fit in the available HBM. In standard embedding
workloads and SC offloaded collectives extremely large per-core partitions
or incorrect sharding specifications can force the compiler to request massive
buffers.
**Recommended actions:**
- **Check sharding:** Ensure your embedding tables and SC input/output tensors
are partitioned/sharded correctly. If a single core is responsible for too
much data, the allocation might fail.
- **Adjust limits:** Review `max_ids_per_partition` and
`max_unique_ids_per_partition`. If these are set unnecessarily high, the
compiler reserves more memory than needed. Refer to
[How limits translate to tables](https://openxla.org/xla/sparsecore#7_how_limits_translate_to_tables_on_sparsecore).
### Scenario 2. Internal memory failures (Memory Space IDs: 0, 201, 202, 204)
Allocation failures in **Smem**, **TileSpmem**, **Spmem**, or **Sync Flags**
typically occur due to compiler bugs or limitations in the allocation strategy,
where the compiler fails to account for all memory
requirements.
**Recommended actions:**
1. **Isolate the failing XLA operation:** To identify the specific SC HLO or
Mosaic kernel causing the failure, generate the intermediate compiler
representations:
- **Dump SparseCore MLIR:** Set the flag
`--xla_sc_dump_mlir_to=/path/to/dump`. This generates the MLIR of the
SparseCore program, allowing you to see which allocation size matches
the error message.
- **Dump Mosaic LLO:** For custom kernels, use
`--xla_mosaic_dump_to=/path/to/dump` to inspect all Low Level Optimizer
(LLO) programs emitted by Mosaic.
2. **Reduce scratch sizes (Pallas users):** If the failure occurs within a
Mosaic kernel, review your `scratch_shapes` configuration. Ensure that your
`pltpu.SMEM` requests fit within the hardware specifications for your
specific TPU generation.
3. **Disable collective offload:** If the error arises from a SC offloaded
collective operations, try disabling the SC offloading features:
- `--xla_tpu_enable_sparse_core_collective_offload_all_gather=false`
- `--xla_tpu_enable_sparse_core_collective_offload_all_reduce=false`
4. **File a bug:** If the above steps do not resolve the issue, it is likely a
compiler bug. [Please file a bug report](https://github.com/openxla/xla).
third_party/xla/docs/errors/error_3001.md
Vendored
+62
View File
@@ -0,0 +1,62 @@
# Error code: E3001
**Category:** CompileTime: SparseCore No Viable Logical Replica Count
This error occurs when the `XLA:SparseCore` compiler fails to determine a
valid logical replica count configuration that allows the workload to fit within
the SparseCore's local scratchpad memory (Tilespmem).
**Sample error messages:**
```
XLA:TPU compile permanent error. Compilation failure: No viable logical replica count for the embedding table with metadata: max_nz_per_row = 141352, max_unique_nz_per_row = 8, feature_width = 8, sample_count = 204800 (last tried split factor for vector splitting = 1, last tried split factor for sample dimension splitting = 1, fixed_size_allocation_bytes = 410880, row_dependent_size_allocation_bytes = 1696224, total_spmem_size_bytes = 524288) ...
```
**XLA backends:** TPU
## Overview
This error is specific to **SparseCore** use cases, particularly for Large
Embedding Models (LEMs).
The **logical replica count** is an internal compiler parameter that determines
how input batches are partitioned to manage scratchpad allocation pressure. The
compiler attempts to split the workload into smaller chunks (replicas) so that
the intermediate buffers required for each chunk fit into the SparseCore's
limited **Scratchpad Memory**. Generally, a higher logical replica count reduces
allocation pressure by processing smaller batches of data at a time.
This error indicates that even after attempting various splitting
configurations, the compiler could not find a setup where the required buffers
fit in the Tilespmem memory. The allocation size is determined by a combination
of:
- **`sample_count`**: The number of embedding lookup IDs assigned to each
SparseCore (derived from batch size).
- **`feature_width`**: The size of the embedding dimension.
- **`max_nz_per_row`**: The maximum number of non-unique embedding lookup IDs
across all SparseCores.
- **`max_unique_nz_per_row`**: The maximum number of unique embedding lookup
IDs.
## Debugging
To resolve this error, you need to reduce the memory pressure on the SparseCore
scratchpad.
### 1. Improve metadata estimations
The compiler allocates memory based on `max_nz_per_row` and
`max_unique_nz_per_row`. If these values are estimated conservatively (i.e., set
much higher than the actual data requires), the compiler will reserve
unnecessary space, causing this error. Ensure these parameters accurately
reflect the actual ID distribution of your dataset.
You can consider applying **Feedback-Directed Optimization (FDO)** to determine
optimal values for these parameters.
### 2. Reduce batch size
The `sample_count` is directly derived from your global batch size. Reducing the
batch size decreases the amount of data each SparseCore must process per step,
thereby reducing the size of the required scratchpad buffers.
third_party/xla/docs/errors_overview.md
Vendored
+36
View File
@@ -0,0 +1,36 @@
# XLA Errors Overview
XLA errors are categorized into different XLA error sources. Each source has a
list of an additional context other than the error message, which will be
attached to each error within the category.
🚧 Note that this standarization effort is a work in progress so not all error
messages will have an attached error code yet.
An example error log might look like:
```
XlaRuntimeError: RESOURCE_EXHAUSTED: XLA:TPU compile permanent error. Ran out of memory in memory space hbm. Used 49.34G of 32.00G hbm. Exceeded hbm capacity by 17.34G. Total hbm usage >= 49.34G: reserved 3.12M program unknown size arguments 49.34G
JaxRuntimeError: RESOURCE_EXHAUSTED: Ran out of memory in memory space vmem while allocating on stack for %ragged_latency_optimized_all_gather_lhs_contracting_gated_matmul_kernel.18 = bf16[2048,4096]{1,0:T(8,128)(2,1)} custom-call(%get-tuple-element.18273, %get-tuple-element.18274, %get-tuple-element.18275, %get-tuple-element.18276, %get-tuple-element.18277, /*index=5*/%bitcast.8695, %get-tuple-element.19201, %get-tuple-element.19202, %get-tuple-element.19203, %get-tuple-element.19204), custom_call_target=""
```
## Statuses and CHECK failures
In general, in XLA we can flag corrupted execution with two mechanisms: statuses
and CHECK macro failures.
Statuses are meant for non-fatal, recoverable errors. The assumption is that the
function returns, and execution continues down the path where the caller
explicitly checks the returned Status object. It's useful for handling invalid
user input or expected resource constraints.
On the other hand, CHECK failures cover programmer's errors or violations of
invariants that should never happen if the code is correct. In case of an
activated CHECK the program will log the error message and immediately
terminate. It could ensure internal consistency, such as checking that a pointer
is non-null before dereferencing it.
## Error codes
Here is an index list with all [error codes](error_codes.md).
third_party/xla/docs/flags_guidance.md
Vendored
+106
View File
@@ -0,0 +1,106 @@
# XLA Flags Guidance
This guide offers a curated selection of key XLA flags to assist users
in effectively navigating and utilizing XLA's capabilities. The following
sections detail flags that can significantly impact runtime performance and
memory utilization. Should any issues, such as crashes, arise after enabling a
flag, it is recommended to revert to the default setting and create a
GitHub issue.
## Correctness Flags
Flag | Description | Default Values | Suggested Values | Candidate Values
:------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :----------------------------------- | :----------------------------------- | :---------------
`xla_mosaic_on_device_checks` | This flag enables on-device checks for Mosaic codegen. Currently, the supported checks are on bounds, i.e., if an out-of-bounds memory is touched, the compilation/execution would catch it. | `xla_mosaic_on_device_checks=bounds` | `xla_mosaic_on_device_checks=bounds` | `xla_mosaic_on_device_checks=bounds`
## Performance Flags
The following flags are instrumental in enhancing runtime performance.
Experimenting with these settings may lead to considerable performance gains.
Flag | Description | Default Values | Suggested Values | Candidate Values
:----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :---------------
**Pipelining** <br> 1. `xla_should_allow_loop_variant_parameter_in_chain` <br> 2. `xla_should_add_loop_invariant_op_in_chain` <br> 3. `xla_tpu_enable_ici_ag_pipelining` | These 3 flags should be used in conjunction to enable collective pipelining of ICI(Interchip-Interconnect) all-gather operations, which creates more opportunities for overlapping execution. | 1. `xla_should_allow_loop_variant_parameter_in_chain=kDisabled` <br> 2. `xla_should_add_loop_invariant_op_in_chain=kDisabled` <br> 3. `xla_tpu_enable_ici_ag_pipelining=false` | 1. `xla_should_allow_loop_variant_parameter_in_chain=kEnabled` <br> 2. `xla_should_add_loop_invariant_op_in_chain=kEnabled` <br> 3. `xla_tpu_enable_ici_ag_pipelining=true` | 1. `xla_should_allow_loop_variant_parameter_in_chain=kDisabled/kEnabled/kAuto` <br> 2. `xla_should_add_loop_invariant_op_in_chain=kDisabled/kEnabled/kAuto` <br> 3. `xla_tpu_enable_ici_ag_pipelining=true/false`
**v5e/Async** <br> `xla_enable_async_all_gather` <br> `xla_tpu_enable_async_collective_fusion` <br> `xla_tpu_enable_async_collective_fusion_fuse_all_gather` | These 3 flags should be used in conjunction to activate asynchronous all-gather operations on v5e. | `xla_enable_async_all_gather=kAuto` <br> `xla_tpu_enable_async_collective_fusion=true` <br> `xla_tpu_enable_async_collective_fusion_fuse_all_gather=true` | `xla_enable_async_all_gather=kAuto` <br> `xla_tpu_enable_async_collective_fusion=true` <br> `xla_tpu_enable_async_collective_fusion_fuse_all_gather=true` | `xla_enable_async_all_gather=kDisabled/kEnabled/kAuto` <br> `xla_tpu_enable_async_collective_fusion=true/false` <br> `xla_tpu_enable_async_collective_fusion_fuse_all_gather=true/false`
**v5e/Async** <br> `xla_tpu_enable_async_collective_fusion` <br> `xla_tpu_enable_async_collective_fusion_fuse_all_reduce` | These 2 flags should be used in conjunction to activate asynchronous all-reduce operations on v5e. | `xla_tpu_enable_async_collective_fusion=true` <br> `xla_tpu_enable_async_collective_fusion_fuse_all_reduce=false` | `xla_tpu_enable_async_collective_fusion=true` <br> `xla_tpu_enable_async_collective_fusion_fuse_all_reduce=true` | `xla_tpu_enable_async_collective_fusion=true/false` <br> `xla_tpu_enable_async_collective_fusion_fuse_all_reduce=true/false`
**Async** <br> `xla_tpu_enable_async_all_to_all` | This flag enables asynchronous all-to-all communication. | `xla_tpu_enable_async_all_to_all=false` | `xla_tpu_enable_async_all_to_all=true` | `xla_tpu_enable_async_all_to_all=true/false`
**Latency-bound** <br> `xla_all_gather_latency_bound_threshold_in_bytes` | This flag is intended for latency-bound (i.e., small-sized) all-gather operations. Enabling this triggers specific optimizations that can reduce execution time for latency-bound all-gathers. Typically it’s used in inference workloads. | `xla_all_gather_latency_bound_threshold_in_bytes=-1` <br> (which is not enabled) | `4~16Mb(i.e. 4~16 * 1024 * 1024)` | `[0, 9223372036854775807]`
**Latency-bound** <br> `xla_all_reduce_latency_bound_threshold_in_bytes` | This flag is intended for latency-bound (i.e., small-sized) all-gather operations. Enabling this triggers specific optimizations that can reduce execution time for latency-bound all-reduces. Typically it’s used in inference workloads. | `xla_all_reduce_latency_bound_threshold_in_bytes=-1` <br> (which is not enabled) | `4~16Mb(i.e. 4~16 * 1024 * 1024)` | `[0, 9223372036854775807]`
**Latency-bound** <br> `xla_collective_permute_latency_bound_threshold_in_bytes` | This flag is intended for latency-bound (i.e., small-sized) all-gather operations. Enabling this triggers specific optimizations that can reduce execution time for latency-bound collective-permutes. Typically it’s used in inference workloads. | `xla_collective_permute_latency_bound_threshold_in_bytes=-1` <br> (which is not enabled) | `4~16Mb(i.e. 4~16 * 1024 * 1024)` | `[0, 9223372036854775807]`
**Latency-bound** <br> `xla_all_to_all_latency_bound_threshold_in_bytes` | This flag is intended for latency-bound (i.e., small-sized) all-gather operations. Enabling this triggers specific optimizations that can reduce execution time for latency-bound all-to-all. Typically it’s used in inference workloads. | `xla_all_to_all_latency_bound_threshold_in_bytes=-1` <br> (which is not enabled) | `4~16Mb(i.e. 4~16 * 1024 * 1024)` | `[0, 9223372036854775807]`
`xla_enable_async_collective_permute` | Rewrites all collective-permute operations to their asynchronous variants. When set to `auto`, XLA can turn on async collective based on other configurations or conditions automatically. | `xla_enable_async_collective_permute=kAuto` | `xla_enable_async_collective_permute=kAuto` | `xla_enable_async_collective_permute=kAuto/kEnabled/kDisabled`
**Compute centric** <br> `xla_tpu_enable_dot_strength_reduction` | This flag rewrites non-compute intensive dots as multiply + reduce operations. | **Compute centric** <br> `xla_tpu_enable_dot_strength_reduction=true` | `xla_tpu_enable_dot_strength_reduction=true` | `xla_tpu_enable_dot_strength_reduction=true/false`
**Compute centric** <br> `xla_tpu_dot_dot_fusion` | This flag enables dot-dot fusion, which fuses a producer-dot operation with a consumer-dot operation. On doing so, the producer-dot's output is not manifested in slow/main memory driving down memory footprint. | `xla_tpu_dot_dot_fusion=true` | `xla_tpu_dot_dot_fusion=true` | `xla_tpu_dot_dot_fusion=true/false`
**Compute centric** <br> `xla_jf_enable_multi_output_fusion` | This flag enables fusions that fuse multiple consumers (i.e. the resultant fusion will have multiple outputs) | `xla_jf_enable_multi_output_fusion=true` | `xla_jf_enable_multi_output_fusion=true` | `xla_jf_enable_multi_output_fusion=true/false`
**Compute centric** <br> `xla_tpu_scoped_vmem_limit_kib` | This flag sets the amount of scratchpad VMEM available to per op for local usage in KiloBytes. Rest of the VMEM is used as buffer space. | `xla_tpu_scoped_vmem_limit_kib=16384` | `xla_tpu_scoped_vmem_limit_kib=16384` | `xla_tpu_scoped_vmem_limit_kib=[4096, VMEM size of the architecture - 1024]`
**Compute centric** <br> `xla_tpu_async_copy_bandwidth_scaling_factor` | Scales effective bandwidth for async copies. This is used when making prefetch decisions and deciding which tensors should live in VMEM. | `xla_tpu_async_copy_bandwidth_scaling_factor=1` | `xla_tpu_async_copy_bandwidth_scaling_factor=1` | `xla_tpu_async_copy_bandwidth_scaling_factor=(0, 1]`
**Compute centric** <br> `xla_msa_enable_cross_program_prefetch_freeing` | Enables freeing optimization for cross-program-prefetched buffers. | `xla_msa_enable_cross_program_prefetch_freeing=enabled` | `xla_msa_enable_cross_program_prefetch_freeing=enabled` | `xla_msa_enable_cross_program_prefetch_freeing=enabled/disabled`
**Compute centric** <br> `xla_tpu_msa_inefficient_use_to_copy_ratio` | The ratio of use bytes to copy bytes for a given allocation site below which we consider the site to be inefficient. This is used while making VMEM placement decisions. A value of 0 would treat all sites as efficient and a value of 1 would require the amount of bytes used at the site to be at least as much as the async copy bytes. | `xla_tpu_msa_inefficient_use_to_copy_ratio=0.5` | `xla_tpu_msa_inefficient_use_to_copy_ratio=0.5` | `xla_tpu_msa_inefficient_use_to_copy_ratio=[0, 1]`
### CPU Performance Flags
Flag | Description | Default Values | Suggested Values | Candidate Values
:--- | :--- | :--- | :--- | :---
`xla_cpu_opt_preset` | Sets the CPU optimization preset. `FAST_COMPILE` packages up a series of compiler tweaks that trade off small amounts of runtime performance for large returns in compile time. `FAST_RUNTIME` is the default and doesn't need to be specified. | `FAST_RUNTIME` | `FAST_COMPILE` (for development) | `FAST_RUNTIME`, `FAST_COMPILE`
## Memory Flags
The flags listed below are provided to address HBM-related issues. These
should only be adjusted if you encounter HBM "out of memory" errors during model
compilation. In all other scenarios, the default values are recommended, as
altering them could adversely affect performance.
Flag | Description | Default Values | Suggested Values | Candidate Values
:------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :----------------------------------------------------- | :------------------------------------------------------ | :---------------
**Scheduler** <br> `xla_latency_hiding_scheduler_rerun` | This setting adjusts the behavior of the latency-hiding scheduler. It works by incrementally reducing the memory limit allocated for scheduling with each "rerun" of the process. | `xla_latency_hiding_scheduler_rerun=1` | `xla_latency_hiding_scheduler_rerun=5` | `0~10(it doesn’t make much sense beyond 10 reruns)`
**Fusion** <br> `xla_tpu_rwb_fusion` | This flag enables reduce+broadcast type of fusions, and may decrease memory usage. | `xla_tpu_rwb_fusion=true` | `xla_tpu_rwb_fusion=false` | `xla_tpu_rwb_fusion=true/false`
**Scheduler** <br> `xla_memory_scheduler` | This flag specifies the algorithm the memory scheduler will use to minimize memory consumption. Using a more advanced algorithm might get a less memory-consuming schedule, at the cost of longer compilation time. | `xla_memory_scheduler=kDefault` | `xla_memory_scheduler=kBrkga` | `xla_memory_scheduler=kDefault/kList/kDfs/kPostOrder/kBrkga`
**Scheduler** <br> `xla_tpu_enable_latency_hiding_scheduler` | This flag enables the latency-hiding scheduler, which allows us to perform asynchronous collective instead of synchronous ones. Disabling it reduces memory usage at the cost of losing the performance gains from these asynchronous operations. | `xla_tpu_enable_latency_hiding_scheduler=true` | `xla_tpu_enable_latency_hiding_scheduler=false` | `xla_tpu_enable_latency_hiding_scheduler=true/false`
**SPMD** <br> `xla_jf_spmd_threshold_for_windowed_einsum_mib` | This flag sets the lower threshold of the minimum size of the dot to trigger collective matmul. Setting it to a higher value would save memory at the cost of losing opportunities to perform collective matmul. | `xla_jf_spmd_threshold_for_windowed_einsum_mib=-1` | `10Mb~1Gb (i.e. 10*1024*1024 ~ 1024*1024*1024)` | `[0, 9223372036854775807]`
**Scheduler** <br> `xla_gpu_enable_analytical_sol_latency_estimator` | This flag enables the analytical estimator which maximizes compute-communication overlap on GPUs. | `xla_gpu_enable_analytical_sol_latency_estimator=true` | `xla_gpu_enable_analytical_sol_latency_estimator=false` | `true/false`
## Other commonly used flags
| Flag | Type | Notes |
| :---- | :---- | :----- |
| `xla_dump_to` | String (filepath) | The folder where pre-optimization HLO files and other artifacts will be placed (see [XLA Tools](https://openxla.org/xla/tools)). |
### TPU XLA flags
Flag | Type | Notes
:---------------------------------------------- | :------------------- | :----
`xla_tpu_enable_data_parallel_all_reduce_opt` | Boolean (true/false) | Optimization to increase overlap opportunities for DCN (data center networking) all-reduces used for data parallel sharding.
`xla_tpu_data_parallel_opt_different_sized_ops` | Boolean (true/false) | Enables pipelining of data parallel ops across multiple iterations even if their output sizes don't match what can be saved in place in the stacked variables. Can increase memory pressure.
`xla_tpu_spmd_rng_bit_generator_unsafe` | Boolean (true/false) | Whether to run RngBitGenerator HLO in a partitioned way, which is unsafe if deterministic results are expected with different shardings on different parts of the computation.
`xla_tpu_megacore_fusion_allow_ags` | Boolean (true/false) | Allows fusing all-gathers with convolutions/all-reduces.
`xla_tpu_enable_ag_backward_pipelining` | Boolean (true/false) | Pipelines all-gathers (currently megascale all-gathers) backwards through scan loops.
### GPU XLA flags
The `-O1` optimization level enables advanced compiler passes for improved GPU
performance, including several categories of flags below: pipelining of
data-parallel collectives (`xla_gpu_enable_pipelined_all_gather`,
`xla_gpu_enable_pipelined_all_reduce`,
`xla_gpu_enable_pipelined_reduce_scatter`), while loop unrolling
(`xla_gpu_enable_while_loop_double_buffering`), latency hiding scheduling
(`xla_gpu_enable_latency_hiding_scheduler`), and SOL latency estimator on
Hopper/Blackwell (`xla_gpu_enable_analytical_sol_latency_estimator`). See
[GPU Effort Levels](https://openxla.org/xla/effort_levels) for details.
Flag | Type | Notes
:------------------------------------------------ | :--------------------------- | :----
`xla_gpu_enable_latency_hiding_scheduler` | Boolean (true/false) | This flag enables latency hiding schedulers to overlap asynchronous communication with computation efficiently. The default value is False.
`xla_gpu_enable_analytical_sol_latency_estimator` | Boolean (true/false) | Enables platform specific scheduling decisions, which in turn improve compute-communication overlap. The default value is true.
`xla_gpu_analytical_latency_estimator_options` | Structured string | Configures parameters for the `xla_gpu_enable_analytical_sol_latency_estimator`. Adjust by setting `nic_speed_gbps=$NIC_SPEED,nccl_op_launch_us=$LAUNCH_OVERHEAD,chunk_prep_us=$CHUNK_PREP,rtt_us=$RTT,chunk_size_bytes=$CHUNK_SIZE,gpus_per_node=$GPUS_PER_NODE`. The default value depends on a detected platform.
`xla_gpu_enable_triton_gemm` | Boolean (true/false) | Use Triton-based matrix multiplication.
`xla_gpu_enable_command_buffer` | List of CommandBufferCmdType | Which kind of commands should be captured in command buffers.
`xla_gpu_all_reduce_combine_threshold_bytes` | Integer (bytes) | These flags tune when to combine multiple small AllGather / ReduceScatter / AllReduce into one big AllGather / ReduceScatter / AllReduce to reduce time spent on cross-device communication. For example, for the AllGather / ReduceScatter thresholds on a Transformer-based workload, consider tuning them high enough so as to combine at least a Transformer Layer’s weight AllGather / ReduceScatter. By default, the combine_threshold_bytes is set to 256.
`xla_gpu_all_gather_combine_threshold_bytes` | Integer (bytes) | See xla_gpu_all_reduce_combine_threshold_bytes above.
`xla_gpu_reduce_scatter_combine_threshold_bytes` | Integer (bytes) | See xla_gpu_all_reduce_combine_threshold_bytes above.
`xla_gpu_enable_pipelined_all_gather` | Boolean (true/false) | Enable pipelinling of all-gather instructions.
`xla_gpu_enable_pipelined_reduce_scatter` | Boolean (true/false) | Enable pipelinling of reduce-scatter instructions.
`xla_gpu_enable_pipelined_all_reduce` | Boolean (true/false) | Enable pipelinling of all-reduce instructions.
`xla_gpu_enable_pipelined_host_offloading` | Boolean (true/false) | Enable pipelining of host offloading instructions.
`xla_gpu_enable_while_loop_double_buffering` | Boolean (true/false) | Enable double-buffering for while loop.
`xla_gpu_enable_all_gather_combine_by_dim` | Boolean (true/false) | Combine all-gather ops with the same gather dimension or irrespective of their dimension.
`xla_gpu_enable_reduce_scatter_combine_by_dim` | Boolean (true/false) | Combine reduce-scatter ops with the same dimension or irrespective of their dimension.
third_party/xla/docs/gpu_architecture.md
Vendored
+253
View File
@@ -0,0 +1,253 @@
# XLA:GPU Architecture Overview
# Introduction
XLA is a hardware- and framework- domain-specific compiler for linear algebra,
offering best-in-class performance. JAX, TF, Pytorch and others use XLA by
converting the user input to
[StableHLO](https://github.com/openxla/stablehlo/tree/main) (“high-level
operation”: a set of \~100 statically shaped instructions like addition,
subtraction, matmul, etc) operation set, from which XLA produces optimized code
for a variety of backends:
![](./images/xla_hardware.png)
During the execution, the frameworks invoke the
[PJRT runtime](https://opensource.googleblog.com/2023/05/pjrt-simplifying-ml-hardware-and-framework-integration.html)
API, which lets the frameworks perform the operation “populate the specified
buffers using a given StableHLO program on a specific device”.
# XLA:GPU Pipeline
XLA:GPU uses a combination of “native” (PTX, via LLVM) emitters and TritonIR
emitters to generate high-performance GPU kernels (blue color indicates 3P
components):
![](./images/gpu_pipeline.png)
## Running Example: JAX
To illustrate the pipeline, let’s start with a running example in JAX, which
computes a matmul combined with multiplication by a constant and negation:
```
def f(a, b):
    return -((a @ b) * 0.125)
```
We can inspect the HLO generated by the function:
```
M = 1024
K = 512
N = 2048
key = jax.random.PRNGKey(1701)
a = jax.random.randint(key, (M, K), dtype=jax.numpy.int8, minval=0, maxval=255)
b = jax.random.normal(key, (K, N), dtype=jax.dtypes.bfloat16)
print(jax.xla_computation(f)(a, b).as_hlo_text())
```
which generates:
```
HloModule xla_computation_f, entry_computation_layout={(s8[1024,512]{1,0}, bf16[512,2048]{1,0})->(bf16[1024,2048]{1,0})}
ENTRY main.10 {
  Arg_0.1 = s8[1024,512]{1,0} parameter(0)
  convert.5 = bf16[1024,512]{1,0} convert(Arg_0.1)
  Arg_1.2 = bf16[512,2048]{1,0} parameter(1)
  dot.6 = bf16[1024,2048]{1,0} dot(convert.5, Arg_1.2), lhs_contracting_dims={1}, rhs_contracting_dims={0}
  constant.3 = bf16[] constant(0.125)
  broadcast.4 = bf16[1024,2048]{1,0} broadcast(constant.3), dimensions={}
  multiply.7 = bf16[1024,2048]{1,0} multiply(dot.6, broadcast.4)
  ROOT negate.8 = bf16[1024,2048]{1,0} negate(multiply.7)
}
```
We can visualize the input HLO computation as well, using
`jax.xla_computation(f)(a, b).as_hlo_dot_graph()`:
![](./images/lowered_hlo.png)
## Optimizations on HLO: Key Components
A number of notable optimization passes happen on HLO, as HLO->HLO rewrites.
### SPMD Partitioner
The XLA SPMD partitioner, as described in
[GSPMD: General and Scalable Parallelization for MLComputation Graphs](https://arxiv.org/pdf/2105.04663.pdf),
consumes HLO with sharding annotations (produced e.g. by `jax.pjit`), and
produces a sharded HLO which can then run on a number of hosts and devices.
Apart from partitioning, the SPMD attempts to optimize HLO for an optimal
execution schedule,
[overlapping](https://dl.acm.org/doi/pdf/10.1145/3567955.3567959) computation
and communication between the nodes.
#### Example
Consider starting from a simple JAX program sharded across two devices:
```
# Defines a mesh with two axes called ‘x’ and ‘y’,
# sharded across two devices: first and second CPU.
with jax.sharding.Mesh(
      [['cpu:0', 'cpu:1']], ('x', 'y')):
    @pjit
    def f(a, b):
        out = -((a @ b) * 0.125)
        # Shard output matrix access across ‘x’
        # and ‘y’ respectively. Generates ‘Sharding’
        # custom call.
        out = with_sharding_constraint(
          out, jax.lax.PartitionSpec('x', 'y'))
        return out
# Random inputs to call our function.
a = jax.random.randint(key, (1024, 512), jnp.int8)
b = jax.random.normal(key, (512, 2048), jnp.float32)
print(f.lower(a, b).compiler_ir())
```
Visualizing it, the sharding annotations are presented as custom calls:
![](./images/annotated_module.png)
To check how the SPMD partitioner expands the custom call, we can look at HLO
after optimizations:
```
print(f.lower(np.ones((8, 8)).compile().as_text())
```
Which generates HLO with a collective:
![](./images/partitioned_module.png)
### Layout Assignment
HLO decouples logical shape and physical layout (how tensors are laid out in
memory). For example, a matrix `f32[32, 64]` can be represented either in
row-major or column-major order, represented as `{1,0}` or `{0,1}` respectively.
In general, layout is represented as a part of shape, showing a permutation over
the number of dimensions indicating physical layout in memory.
For each operation present in the HLO, the Layout Assignment pass chooses an
optimal layout (e.g. NHWC for a convolution on Ampere). For example, an
`int8xint8->int32`  matmul operation prefers `{0,1}` layout for the RHS of the
computation. Similarly, “transposes” inserted by the user are ignored, and
encoded as a layout change.
The layouts are then propagated through the graph, and conflicts between layouts
or at graph endpoints are materialized as `copy` operations, which perform the
physical transposition. For example, starting from the graph
![](./images/pre_layout_module.png)
Running the layout assignment we see the following layouts and `copy` operation
inserted:
![](./images/layout_assigned_module.png)
### Fusion
Fusion is XLA’s single most important optimization, which groups multiple
operations (e.g. addition into exponentiation into matmul) to a single kernel.
Since many GPU workloads tend to be memory-bound, fusion dramatically speeds up
the execution by avoiding the writing of intermediate tensors to HBM and then
reading them back, and instead passes them around in either registers or shared
memory.
Fused HLO instructions are blocked together in a single fusion computation,
which establishes the following invariants:
- No intermediate storage inside the fusion is materialized in HBM (it has to
be all passed through either registers or shared memory).
- A fusion is always compiled to exactly one GPU kernel
## HLO Optimizations on Running Example
We can inspect the post-optimization HLO using `jax.jit(f).lower(a,
b).compile().as_text()`, and verify that a single fusion got generated:
```
HloModule jit_f, is_scheduled=true, entry_computation_layout={(s8[3,2]{1,0}, bf16[2,3]{1,0})->bf16[3,3]{1,0}}, allow_spmd_sharding_propagation_to_output={true}
%triton_gemm_dot.6_computation (parameter_0: s8[3,2], parameter_1: bf16[2,3]) -> bf16[3,3] {
  %parameter_0 = s8[3,2]{1,0} parameter(0)
  %convert.0 = bf16[3,2]{1,0} convert(s8[3,2]{1,0} %parameter_0)
  %parameter_1 = bf16[2,3]{1,0} parameter(1)
  %dot.0 = bf16[3,3]{1,0} dot(bf16[3,2]{1,0} %convert.0, bf16[2,3]{1,0} %parameter_1), lhs_contracting_dims={1}, rhs_contracting_dims={0}
  %convert.1 = f32[3,3]{1,0} convert(bf16[3,3]{1,0} %dot.0)
  %constant_0 = bf16[] constant(0.125)
  %broadcast.0 = bf16[3,3]{1,0} broadcast(bf16[] %constant_0), dimensions={}
  %convert.2 = f32[3,3]{1,0} convert(bf16[3,3]{1,0} %broadcast.0)
  %multiply.0 = f32[3,3]{1,0} multiply(f32[3,3]{1,0} %convert.1, f32[3,3]{1,0} %convert.2)
  %negate.0 = f32[3,3]{1,0} negate(f32[3,3]{1,0} %multiply.0)
  ROOT %convert.6 = bf16[3,3]{1,0} convert(f32[3,3]{1,0} %negate.0)
}
ENTRY %main.9 (Arg_0.1: s8[3,2], Arg_1.2: bf16[2,3]) -> bf16[3,3] {
  %Arg_1.2 = bf16[2,3]{1,0} parameter(1), sharding={replicated}
  %Arg_0.1 = s8[3,2]{1,0} parameter(0), sharding={replicated}
  ROOT %triton_gemm_dot.6 = bf16[3,3]{1,0} fusion(s8[3,2]{1,0} %Arg_0.1, bf16[2,3]{1,0} %Arg_1.2), kind=kCustom, calls=%triton_gemm_dot.6_computation, backend_config={"kind":"__triton_gemm","triton_gemm_config":{"block_m":"64","block_n":"64","block_k":"64","split_k":"1","num_stages":"2","num_warps":"4"}}
}
```
Note that the fusion `backend_config` tells us that Triton will be used as a
code generation strategy, and it specifies the chosen tiling.
We can also visualize the resulting module:
![](./images/fused_module.png)
## Buffer Assignment and Scheduling
A buffer assignment pass takes into account the shape information, and aims to
produce an optimal buffer allocation for the program, minimizing the amount of
intermediate memory consumed. Unlike TF or PyTorch immediate-mode (non-compiled)
execution, where the memory allocator does not know the graph in advance, the
XLA scheduler can “look into the future” and produce an optimal computation
schedule.
## Compiler Backend: Codegen and Library Selection
For every HLO instruction in the computation, XLA chooses whether to run it
using a library linked into a runtime, or to codegen it to PTX.
### Library Selection
For many common operations, XLA:GPU uses fast-performance libraries from NVIDIA,
such as cuBLAS, cuDNN, and NCCL. The libraries have an advantage of verified
fast performance, but often preclude complex fusion opportunities.
### Direct code generation
The XLA:GPU backend generates high-performance LLVM IR directly for a number of
operations (reductions, transposes, etc).
### Triton code generation
For more advanced fusions which include matrix multiplication or softmax,
XLA:GPU uses [Triton](https://github.com/openai/triton) as a code-generation
layer. HLO Fusions are converted to TritonIR (an MLIR dialect which serves as an
input to Triton), selects tiling parameters and invokes Triton for PTX
generation:
![](./images/triton_opt_pipeline.png)
We have observed the resulting code to perform very well on Ampere, at
near-roofline performance with properly tuned tile sizes.
## Runtime
XLA Runtime converts the resulting sequence of CUDA kernel calls and library
invocations into a RuntimeIR (an MLIR dialect in XLA), on which CUDA graph
extraction is performed. CUDA graph is still work in progress, only some nodes
are currently supported. Once CUDA graph boundaries are extracted, RuntimeIR is
compiled via LLVM to a CPU executable, which can then be stored or transferred
for Ahead-Of-Time compilation.
third_party/xla/docs/hlo_dumps.md
Vendored
+269
View File
@@ -0,0 +1,269 @@
# Dump HLO Computations
An HLO dump is a textual representation of the HLO modules at different stages
of the computation. It is useful for debugging, and you often need to include it
in bug reports. This is typically a human-readable **text file** that lists the
HLO instructions and their properties. Sometimes, HLO modules are dumped as:
- **HloProto:** Protocol buffer files, which are a more structured,
machine-readable format.
- **HloSnapshot**: HLO module plus its inputs. For replaying HLOs, you
sometimes require the actual inputs fed to a given computation rather than
random data.
You can use XLA flags to specify and get dumps. In most cases, you can set it
with an environment variable. JAX also offers a programmatic way to print the
HLO dump.
## Local Execution
### Using Environment Variables
You can set the `XLA_FLAGS` environment variable with the necessary flags to get
dumps. This works for JAX, TensorFlow, and PyTorch/XLA.
To dump HLO modules and other debugging information to a specific directory, run
your program with the `--xla_dump_to` flag:
```shell
XLA_FLAGS="--xla_dump_to=DIRECTORY_PATH"
```
For example, you can use `/tmp` or `/tmp/xladump` as the paths.
By default, this dumps HLO modules as text, at the very beginning and end of the
optimization pipeline.
You can also explicitly specify the format:
1. Text dumps
```shell
XLA_FLAGS="--xla_dump_hlo_as_text --xla_dump_to=DIRECTORY_PATH"
```
1. HLO protos
```shell
XLA_FLAGS="--xla_dump_hlo_as_proto --xla_dump_to=DIRECTORY_PATH"
```
1. HLO Snapshots
```shell
XLA_FLAGS="--xla_dump_hlo_snapshots --xla_dump_to=DIRECTORY_PATH"
```
1. Graph render with graphviz server (only works well for small graphs)
```shell
XLA_FLAGS="--xla_dump_hlo_as_url --xla_dump_to=DIRECTORY_PATH"
```
1. Graph render to HTML file (only works well for small graphs)
```shell
XLA_FLAGS="--xla_dump_hlo_as_html --xla_dump_to=DIRECTORY_PATH"
```
For larger graphs, you can use `interactive_graphviz` to visualize parts of the
graph.
**Note:** If `--xla_dump_to` is not specified but another dumping flag is
specified, it will dump to stdout. But the dump will not include binary data,
e.g., proto files, to stdout.
You can also set the HLO Dumps to use syntactic sugar wrappers as op names, by
setting the `--xla_syntax_sugar_async_ops` flag to `true`. This can reduce the
dump by about 20%. By default, this flag is set to `false`, and actual op names
are used in the dump.
```shell
XLA_FLAGS="--xla_dump_to=DIRECTORY_PATH --xla_syntax_sugar_async_ops=true"
```
## Dump Specific Intermediate Passes
In addition to the standard pre-optimized / final-optimized HLOs, you can also
dump the state of HLOs after a particular compiler pass.
```shell
XLA_FLAGS="--xla_dump_hlo_pass_re=regex --xla_dump_to=DIRECTORY_PATH"
```
HLO modules will be dumped for the passes whose names match the regular
expression (regex). For example, you can observe the HLOs resulting from passes
related to SPMD partitioning with:
```shell
XLA_FLAGS="--xla_dump_to=DIRECTORY_PATH --xla_dump_hlo_pass_re=spmd|propagation"
```
To dump the result after every XLA pass (this will result in a lot of files),
you can set:
```shell
XLA_FLAGS="--xla_dump_to=DIRECTORY_PATH --xla_dump_hlo_pass_re=.*"
```
### JAX-specific Options
#### Programmatically in JAX
Instead of passing flags or environment variables, you can also programmatically
dump HLO using JAX’s `lower` and `compile` APIs.
Locally fetch the unoptimized original lowered HLO with:
```python
jax.jit(f).lower(*args).as_text('hlo')
```
For dumping to files during HLO compilation passes, specify:
```python
compilation_args = {
'xla_dump_to': DIRECTORY_PATH,
'xla_dump_hlo_pass_re': 'spmd|propagation', # or some other pass filter
...
}
jax.jit(f).lower(*args).compile(compilation_args)
```
#### Dump jaxprs
[`jaxpr`s](https://docs.jax.dev/en/latest/jaxpr.html) are JAX's intermediate
representation for program traces. To dump this, set the environment variables:
```shell
JAX_DUMP_IR_TO="DIRECTORY_PATH" JAX_DUMP_IR_MODES=jaxpr
```
Learn more in JAX documentation on
[Exporting and serializing staged-out computations: Debugging](https://docs.jax.dev/en/latest/export/export.html#debugging).
## Google Colab
### Environment variables
In the first executed cell of your notebook (because environment variables and
command-line flags are usually only processed once, e.g., at module-import time
or XLA backend initialization time), add the `XLA_FLAGS` detailed above with
`os.environ`, for example:
```python
import os
os.environ['XLA_FLAGS'] = "--xla_dump_to=DIRECTORY_PATH"
```
This will dump the computation to `DIRECTORY_PATH`, for example `/tmp`. On
Colab, navigate to the "Files" browser in the left sidebar, to view and access
this directory.
You can use all the flags mentioned in the Local Execution section.
### JAX-specific options
Similar to local execution; for live, interactive introspection you can directly
print a computation’s pre-optimized HLO:
```python
def f(x):
return jax.numpy.sin(jax.numpy.cos(x))
c = jax.jit(f).lower(3.).compiler_ir('hlo')
print(c.as_hlo_text())
```
You can also directly print a computation’s optimized HLO:
```python
def optimized_HLO(f, *args, platform=None):
print(jax.jit(f).lower(*args).compile().as_text())
def f(x):
return jax.numpy.sin(jax.numpy.cos(x))
optimized_HLO(f, 1.0)
```
#### Dumping All/Small Computations
If you want to see everything in a dump including all small compilations, set
the JAX environment variable:
```shell
JAX_COMPILER_DETAILED_LOGGING_MIN_OPS=0
```
#### Mosaic
Mosaic is a compiler for the Pallas TPU backend, and the experimental Pallas GPU
backend. To dump mosaic computation, set the following flag:
```shell
--xla_mosaic_dump_to=/tmp/mosaic_dumps
```
Or, set TPU init arguments as an environment variable:
```shell
export LIBTPU_INIT_ARGS="--xla_mosaic_dump_to=/tmp/mosaic_dumps"
```
Check out the
[JAX documentation on Pallas and Mosaic](https://docs.jax.dev/en/latest/pallas/index.html)
to learn more.
## More with HLO Dumps
### Finding the right computation
Usually, many computations get dumped. The dumped files are explicitly named
with the JAX, Tensorflow, or PyTorch/XLA "computation name” that are called out
in the logs, making it easy to identify the relevant HLO files. For example:
```
1624325116260738.module_0065.pmap__unnamed_wrapped_function_.186875.before_optimizations.txt
```
Otherwise, you can use `ripgrep` to quickly identify which module holds
particular symbols or computations.
**Tip:** Include the 3 dumped before/after/buffer-assignment files of interest
in your bug reports.
### HLO Conversion
A tool called `hlo-opt` that can translate between HLOProto and text formats.
It's useful in cases where you have one format, but need the other for
debugging.
Learn to use it:
[XLA Tooling documentation: hlo-opt](tools.md#hlo-opt-convert-hlo-module-formats).
### Replay
You can run (replay) the dumped computations on a specified XLA backend with
fake data or input snapshots. This is a convenient way to reproduce, iterate,
and debug issues in XLA.
The following commands use fake data. If you have saved HLO Snapshots, you can
pass those in instead, and the data from the snapshot will be used. To still use
fake data while running the snapshot, pass the flag `--force_fake_data`.
CPU backend:
```shell
bazel run -c opt //xla/hlo/tools:run_hlo_module -- --platform=cpu
/tmp/xladump/module_4561.before_optimizations.txt
```
GPU backend:
```shell
bazel run -c opt //xla/hlo/tools:run_hlo_module -- --platform=CUDA
/tmp/xladump/module_4561.before_optimizations.txt
```
third_party/xla/docs/hlo_isolation.md
Vendored
+200
View File
@@ -0,0 +1,200 @@
# HLO Isolation User Guide
This document explains how to install and use the HLO Isolation API and CLI.
The HLO isolation tooling helps developers and researchers isolate, verify, and
debug numeric mismatches and stability issues in compiled HLO modules.
## Installation
You can use HLO isolation components through the standard OpenXLA/TensorFlow
build mechanism (Bazel) or via binary distribution.
### Source and Bazel Setup
When building the compiler and tools stack from source, include or depend on the
following libraries:
- API Target:
`//third_party/tensorflow/compiler/xla/tools/hlo_isolation:hlo_isolation_api`
- CLI Target:
`//third_party/tensorflow/compiler/xla/tools/hlo_isolation:hlo_isolation_test`
To build the standalone CLI tool:
```bash
bazel build -c opt //third_party/tensorflow/compiler/xla/tools/hlo_isolation:hlo_isolation_test
```
## Command-Line Interface (CLI)
The `hlo_isolation_test` CLI allows you to isolate and run numeric mismatch and
stability checks against compiled HLO modules directly from a terminal. It
compares the execution results across different environments (e.g., TPU vs.
Defused TPU / CPU / Interpreter).
### Flag Reference
The CLI supports the following flags:
- `--hlo_file`: Path to the input `.hlo` or `.pbtxt` file to load. Can be text
or proto format (required).
- `--test_platform`: Target platform to run the primary test on (e.g., `cpu`,
`gpu`, `tpu`). Defaults to `cpu`.
- `--reference_platform`: Reference platform for baseline comparison (e.g.,
`interpreter`). If empty, reference comparison is disabled.
- `--filter_by_name`: Regular expression to match the module name. Only
matching modules will be run. Defaults to `.*`.
- `--skip_by_name`: Regular expression to match the module name. Matching
modules will be skipped.
- `--filter_by_opcode`: Regular expression to match instruction opcodes. Only
modules containing at least one matching opcode will be run. Defaults to
`.*`.
- `--skip_by_opcode`: Regular expression to match instruction opcodes. Modules
containing any matching opcode will be skipped.
- `--abs_error_bound`: Absolute error bound used for comparison. Defaults to
`0.01`.
- `--rel_error_bound`: Relative error bound used for comparison. Defaults to
`0.1`.
- `--run_hlo_passes`: Boolean flag to determine whether to run standard HLO
passes on the submodules. Defaults to `false`.
- `--shard_index`: The specific shard index to run (zero-based). Defaults to
`-1` (disabled).
- `--num_shards`: The total number of shards. Defaults to `1`.
### Basic Invocation
```bash
./hlo_isolation_test \
--hlo_file=/path/to/failing_fusion.hlo \
--test_platform=gpu \
--reference_platform=interpreter
```
## Result and Artifact Dumps
When a submodule encounters a numeric mismatch or other failure during isolation
testing, the tool automatically serializes debug artifacts to disk for deeper
inspection.
### Dump Contents
On numeric mismatch, the tool writes the following debug artifacts:
1. The failed HLO submodule text (`failed-module-<module_name>.txt`).
2. The expected output literal (`failed-<module_name>-expected.txt`).
3. The actual output literal (`failed-<module_name>-actual.txt`).
4. The mismatching elements summary (`failed-<module_name>-mismatches.txt`).
### Dump Target Location
- **Test Environment:** If run via `bazel test` or an environment defining the
`TEST_UNDECLARED_OUTPUTS_DIR` environment variable, the results are placed
directly in that directory with the exact file names listed above (e.g.,
`failed-<module_name>-expected.txt`).
- **Standard/Manual Run:** When executed manually via the command line, the
artifacts are written to the operating system's temporary directory (e.g.,
`/tmp`), preserving the exact same unified file naming conventions (e.g.,
`/tmp/failed-<module_name>-expected.txt`).
## C++ Integration and API
For developers building custom compiler passes, testing rigs, or automated
pipelines, the C++ API provides a direct way to integrate isolation tests.
### Using the API Directly
The core API provides functional interfaces to run modules and fetch structured
reports:
```cpp
#include "third_party/tensorflow/compiler/xla/tools/hlo_isolation/hlo_isolation_api.h"
xla::hlo_isolation::PipelineIsolationOptions options;
options.module_options.abs_error_bound = 0.01;
options.module_options.rel_error_bound = 0.1;
// Filter specific opcodes programmatically
options.filter_by_opcode = "exponential";
absl::StatusOr<std::vector<xla::HloIsolationTestResult>> results =
xla::hlo_isolation::RunIsolationPipeline(
input_hlo_module,
&my_test_runner,
&my_reference_runner,
options);
```
### Using the Test Mixin
When writing GoogleTest C++ test suites, you can inherit from
`HloIsolationTestMixin` for built-in assertion handling. The base class must
provide both a test runner and a reference runner (e.g., via
`HloPjRtInterpreterReferenceMixin`):
```cpp
#include "third_party/tensorflow/compiler/xla/tests/hlo_pjrt_interpreter_reference_mixin.h"
#include "third_party/tensorflow/compiler/xla/tools/hlo_isolation/hlo_isolation_test_base.h"
class MyCustomPassIsolationTest : public xla::hlo_isolation::HloIsolationTestMixin<
xla::HloPjRtInterpreterReferenceMixin<xla::HloPjRtTestBase>> {};
TEST_F(MyCustomPassIsolationTest, ChecksMyFusionSanity) {
RunAndVerifyIsolationTest(my_failing_module);
}
```
## Sharded Execution (K8s/Slurm)
For large modules or heavy test matrices, you can partition execution across
multi-device clusters (such as Google Kubernetes Engine or Slurm) using the
sharding flags. Each `shard_index` deterministically runs an isolated subset of
the decomposed submodules. This allows for reproducible distributed verification
and targeted re-execution of failing partitions.
### Example: Kubernetes Job
Each test shard is executed as a separate Kubernetes pod using `completionMode:
Indexed`. The `JOB_COMPLETION_INDEX` is passed directly to the CLI's
`--shard_index` flag.
```yaml
apiVersion: batch/v1
kind: Job
metadata:
name: hlo-isolation-job
spec:
completions: 50
parallelism: 50
completionMode: Indexed
template:
spec:
containers:
- name: test-runner
image: gcr.io/my-project/hlo-isolation-tools:latest
command: ["/bin/sh", "-c"]
args:
- |
./hlo_isolation_test \
--num_shards=50 \
--shard_index=$JOB_COMPLETION_INDEX \
--hlo_file=/data/path/to/hlo.hlo
volumeMounts:
- name: hlo-data-volume
mountPath: /data
volumes:
- name: hlo-data-volume
csi:
driver: gcsfuse.csi.storage.gke.io
volumeAttributes:
bucketName: my-xla-debug-bucket
```
## Key Capabilities
- **Portability:** Decouples internal test wrappers from the standalone API,
making it easy to debug HLO mismatches locally.
- **Granularity:** Granular opcode and name filtering improves the debugging
loop when interacting with massive HLO dumps.
- **Extensibility:** Custom runner execution callbacks and data injectors
(`make_fake_arguments_fn`) permit full customization for advanced
verification workflows.
third_party/xla/docs/hlo_passes.md
Vendored
+230
View File
@@ -0,0 +1,230 @@
# HLO Passes
This document outlines the [HLO](https://openxla.org/xla/terminology)
optimizations and transformations passes in the
[XLA compiler](https://openxla.org/xla/architecture).
## Introduction
A single HLO Pass can be comprised of one or many compiler optimizations and
transformations, and XLA provides several hundred such passes. HLO focuses only
on the shape (e.g. a 3x4 matrix) and the
[operation semantics](https://openxla.org/xla/operation_semantics) of the arrays
to make the optimization or transformation easier.
For example:
* [`AlgebraicSimplifier`:](https://github.com/openxla/xla/blob/c37fc6a383b870f43cef82280418fcefcc90b0f8/xla/hlo/transforms/simplifiers/algebraic_simplifier.h#L417)
A pass that performs a number of mostly arithmetic simplifications and
optimizations. Including:
* When dividing by a constant, an optimization is performed to transform
the operation to multiplication by the inversion of the constant.
* [`HloRematerialization`:](https://github.com/openxla/xla/tree/main/xla/hlo/transforms/simplifiers/hlo_rematerialization.h)
A pass that recomputes selected expressions in the computation to reduce
memory pressure caused by long live ranges of array-shaped values.
## Developer details
The base class for HLO passes can be found in
[`xla/hlo/pass/hlo_pass_interface.h`](https://github.com/openxla/xla/blob/main/xla/hlo/pass/hlo_pass_interface.h).
HLO pass should not extend this class directly but instead should extend
[`HloModulePass`](https://github.com/openxla/xla/blob/main/xla/hlo/pass/hlo_pass_interface.h#L142).
See also
[XLA HLO Pass Framework](https://github.com/openxla/xla/tree/main/xla/hlo/pass#readme).
### Tooling and Testing
XLA comes with multiple command line tools, including the hlo-opt tool. This
tool allows execution of an individual pass independent of the given platform
compilation stages. For more information see
[Tooling](https://openxla.org/xla/tools#hlo-opt_hlo_pass_development_and_debugging).
For information on writing unit tests for HLO Passes see
[Testing HLO Passes](https://openxla.org/xla/test_hlo_passes).
## Hardware-independent HLO Pass Examples
This section describes a few examples of passes shared across XLA backends. Some
passes may be specialized for specific backends, but the high-level
functionality is similar.
Shared passes or hardware-independent passes can be found in
[`xla/hlo/transforms`](https://github.com/openxla/xla/tree/main/xla/hlo/transforms).
### Rematerialization
See also
[`HloRematerialization`](https://github.com/openxla/xla/blob/main/xla/hlo/transforms/simplifiers/hlo_rematerialization.h).
Selectively recomputes expressions within the HLO graph to reduce memory usage.
Trades off higher compute for lower memory usage. Can reduce memory usage by
tens of percent and is required to run many large models.
### Algebraic Simplifier
See also
[`AlgebraicSimplifier`](https://github.com/openxla/xla/blob/main/xla/hlo/transforms/simplifiers/algebraic_simplifier.h).
A grab bag of simplifications, optimizations, and canonicalizations. Analogous
to
[LLVM’s `instcombine` pass](https://llvm.org/docs/Passes.html#instcombine-combine-redundant-instructions).
### Constant Folding
See also
[`HloConstantFolding`](https://github.com/openxla/xla/blob/main/xla/hlo/transforms/simplifiers/hlo_constant_folding.h).
Replaces expressions which can be evaluated at compile time with their constant
equivalent.
### Dead Code Elimination
See also
[`HloDCE`](https://github.com/openxla/xla/blob/main/xla/hlo/transforms/simplifiers/hlo_dce.h)
.
Removes operations with unused results (fast implementation).
### Call Graph Flattening
See also
[`FlattenCallGraph`](https://github.com/openxla/xla/blob/main/xla/hlo/transforms/simplifiers/flatten_call_graph.h).
A legalization pass which converts the HLO call graph into a tree by cloning
computations. Required because memory is statically assigned to HLO operations
and not based on dynamic call context.
### Reshape Mover
See also
[`ReshapeMover`](https://github.com/openxla/xla/blob/main/xla/hlo/transforms/simplifiers/reshape_mover.h).
Reshapes and transposes can be expensive, especially on TPU. This pass moves and
reshapes and transposes across elementwise operations enabling the operations to
be merged or eliminated.
### Zero-sized HLO Elimination
See also
[`ZeroSizedHloElimination`](https://github.com/openxla/xla/blob/main/xla/hlo/transforms/simplifiers/zero_sized_hlo_elimination.h).
HLO supports arrays of zero size (one or more dimensions has a bound of zero).
This pass simplifies the graph by replacing zero-sized operations with
zero-sized constants.
## TPU-specific HLO Pass Examples
Passes specific to the TPU backend.
### Model parallelism
The partitioning of an XLA program across multiple cores is performed at the HLO
level and the TPU HLO pipeline includes a number of passes for supporting
multi-core execution.
#### Spatial partitioning
See also
[`ShardingPropagation`](https://github.com/openxla/xla/blob/main/xla/service/sharding_propagation.h).
Pass to support dividing operations across devices along non-batch dimensions.
### Handling of bfloat16
See also
[`BFloat16ConversionFolding`](https://github.com/openxla/xla/blob/main/xla/hlo/transforms/simplifiers/bfloat16_conversion_folding.h),
[`BFloat16MixedPrecisionRemoval`](https://github.com/openxla/xla/blob/main/xla/hlo/transforms/simplifiers/float_normalization.h),
and
[`BFloat16Propagation`](https://github.com/openxla/xla/blob/main/xla/hlo/transforms/bfloat16_propagation.h).
TPUs support bfloat16 as a lower-precision, more compact floating-point
representation than 32-bit floats. Using bfloat16 reduces memory footprint and
memory bandwidth. The TPU HLO pipeline includes various passes for replacing
floats with bfloat16 into the program and propagating the precision through the
graph.
### Legalization passes
See also
[`GatherExpander`](https://github.com/openxla/xla/blob/main/xla/service/gather_expander.h),
and
[`BatchNormExpander`](https://github.com/openxla/xla/blob/main/xla/service/batchnorm_expander.h).
Passes which transform unsupported HLO into a form which the backend can emit or
for which the backend produces a more efficient lowering.
## GPU-specific HLO Pass Example
Passes specific to the GPU backend are found in
[`xla/service/gpu`](https://github.com/openxla/xla/tree/main/xla/service/gpu).
These passes can be identified as classes defined in `namespace gpu`.
### cuDNN Rewriter
See also
[`CudnnFusedConvRewriter`](https://github.com/openxla/xla/blob/main/xla/service/gpu/transforms/cudnn_fused_conv_rewriter.h)
and
[`CudnnNormRewriter`](https://github.com/openxla/xla/blob/main/xla/service/gpu/transforms/cudnn_norm_rewriter.h).
Rewrites fused convolution and norm operations into their respective library
calls in cuDNN.
## CPU-specific HLO Pass Examples
Passes specific to the CPU backend are found in
[`xla/service/cpu`](https://github.com/openxla/xla/tree/main/xla/service/cpu).
These passes can be identified as classes defined in `namespace cpu`.
### Convolution Canonicalization
See also
[`ConvCanonicalization`](https://github.com/openxla/xla/blob/main/xla/service/cpu/conv_canonicalization.h).
Canonicalizes convolutions so that they can be lowered to a fast implementation
in Eigen.
### Operation Parallelization
See also
[`ParallelTaskAssigner`](https://github.com/openxla/xla/blob/main/xla/service/cpu/parallel_task_assignment.h).
Partitions HLOs into tasks to run on separate threads.
## Analysis passes
Analysis passes are not considered "HLO passes" since they do not transform HLO
and may not extend `HloModulePass`. Shared analyses are found in
[`xla/hlo/analysis`](https://github.com/openxla/xla/tree/main/xla/hlo/analysis).
### Analysis Pass Examples
#### Dataflow Analysis
See also
[`HloDataflowAnalysis`](https://github.com/openxla/xla/tree/main/xla/hlo/analysis/hlo_dataflow_analysis.h).
Identifies all HLO values in the graph and their uses.
#### Alias Analysis
See also
[`HloAliasAnalysis`](https://github.com/openxla/xla/tree/main/xla/hlo/analysis/hlo_alias_analysis.h).
Identifies must-alias relationships between values in the program.
#### Computation Cost Analysis
See also
[`HloCostAnalysis`](https://github.com/openxla/xla/tree/main/xla/service/hlo_cost_analysis.h).
Computes FLOP count and memory usage for all operations in the program.
#### HLO Verification
See also
[`HloVerifier`](https://github.com/openxla/xla/tree/main/xla/service/hlo_verifier.h).
Verifies various invariants of the HLO graph.
third_party/xla/docs/hlo_to_thunks.md
Vendored
+277
View File
@@ -0,0 +1,277 @@
# From HLO to Thunks
This document outlines the journey of an XLA *High Level Optimizer* (HLO) module
from its initial state to a final executable. Sometimes we will omit the
"module" and refer to it just as "HLO".
<img src="./images/hlo_to_thunks.svg" alt="HLO to thunks diagram" width="50%">
## Pre-optimization HLO
We start with pre-optimization HLO module. Pre-optimization HLO does not contain
operations (*ops*) that are considered internal to XLA, such as `fusion` or
`bitcast`. Ops don't have a layout at this stage, or if they do, it will be
ignored. Pre-optimization HLO is usually produced by higher-level frameworks
like TensorFlow and JAX. When using the XLA flag `-xla_dump_to`, the
pre-optimization HLO is dumped to a file with file name suffix
“before_optimizations.txt”.
## Optimize HLO Module
The XLA:GPU pipeline turns the pre-optimization HLO into optimized HLO by
running a sequence of passes. The passes can be grouped together semantically
and run in the following order:
### Sharding related passes
This includes passes like the [Shardy
Partitioner](https://openxla.org/shardy/overview) or those for SPMD sharding.
### Optimization passes
This can include both legalization passes and simplification passes.
### Collective optimization passes
Similar to **Optimization passes**, but focuses on collective ops.
### Layout assignment passes
Each HLO op is assigned a layout which is a part of the instruction shape. The
layout controls how the tensor is laid out physically in memory.
Example of a shape with layout:
```
f32[10,20,30]{2,0,1}
```
After the element type, there are the logical dimensions of the shape, followed
by the layout permutation in minor to major order. In this example, the most
minor dimension is 30, the second most minor dimension is 10, and the major
dimension is 20.
The goal of layout assignment is to minimize the number of required physical
transpositions using a greedy strategy. It starts with certain layout
constraints (e.g., cuDNN/cuBLAS libraries expect consecutive dimensions) and
propagates layouts “down” and then “up” the HLO graph. At the
end of layout propagation, some instructions may have conflicting layouts, one
propagated from an operand, one propagated from a user. To resolve this
conflict, a `copy` HLO instruction is inserted that changes the layout from the
operand layout to the instruction layout.
### Layout normalization passes
Given that it is somewhat difficult to figure out the physical shape, layout
normalization attempts to rewrite the shape such that it uses the default layout
`{rank-1, rank-2, …, 0}`. In the example above, the normalized shape would be
`f32[20,10,30]{2,1,0}`. Copy ops that change layouts are rewritten as a
combination of `transpose` and `bitcast`. Given that currently we cannot
normalize all ops, there are still some ops that may have non-default layouts,
most notably `gather` and `dot`. At the boundaries between normalized ops and
non-normalized ops there will be `bitcast` ops that represent a transpose, i.e.
a transpose with a layout assigned that makes it a no-op physically.
Layout normalization also makes some implicit transposes explicit which is
important because codegen can handle explicit transposes with a dedicated
emitter. For example, a reshape is technically allowed to have a different
physical layout between operand and result (e.g. due to different rank). The
`ReshapeDecomposer` pass that runs as part of the layout normalization passes
turns a reshape into a sequence of `transpose`, reshape `bitcast` and
`transpose`.
### Post layout assignment optimization passes
The most important passes here are Triton fusions (GEMM fusions +
Softmax/Layernorm fusions) or rewrites to library calls. Autotuning also runs in
this step, where XLA chooses chooses between different emitters, picks the best
algorithm for convolutions or dots, finds the best tiling for fusions handled by
the Triton emitter etc.
### Fusion passes
The two main passes are `PriorityFusion` and `Multi-Output` fusion.
In `PriorityFusion`, we form fusions guided by the cost model. When fusing we
would allow duplicating ops with several users if the op can be fused into all
users. We would also allow extending existing Triton Softmax fusions if
possible.
`Multi-Output` fusion is a separate pass that allows fusing ops/fusions that
share an operand. It can also fuse operands/operand fusions into users without
duplication by adding extra output(s), so other users of the op to be fused can
be redirected to these outputs. This pass needs to be careful not to introduce
cycles into the HLO graph.
After Multi-Output fusion, common subexpression elimination (`HloCSE` pass)
runs, potentially merging previously duplicated ops back together if they ended
up in the same fusion.
### Several post-fusion passes
Several passes related to collectives (like turning them to async, or enforcing
a certain relative order of collectives).
Finally we run `CopyInsertion` where copies are added to ensure that in-place
operations don't overwrite data that is still needed elsewhere.
At the end of optimization, the optimized HLO is dumped if using the flag
`-xla_dump_to` to a file that has the file name suffix
"after_optimizations.txt". If you want to dump the HLO after intermediate
passes that actually change the HloModule, you can use the flag
`-xla_dump_hlo_pass_re=.*` (or a specific regular expression to restrict it to
certain passes).
## Scheduling
An HLO Module without a schedule still has some degree of freedom in the order
in which ops are processed. Any topological sort respecting operand/result
relationships and control dependencies is valid. Scheduling determines what
specific order to use. The main concern at this stage is the maximum memory
consumption that depends on the lifetime of tensors. In an initial step, we try
different scheduler algorithms and pick the schedule that should minimize the
peak memory consumption. Note that at this point we don't work with a physical
buffers yet (that will happen in "Buffer Assignment") and simulate the memory
usage.
Then `LatencyHidingScheduler` pass runs and tries to maximize
compute-communication overlap. But that may increase memory usage again.
Finally, in case peak memory consumption is higher than the amount of memory we
have available we run `HloRematerialization`. This pass attempts to reduce
memory usage at the cost of performance, as e.g. some fusions might be split and
some ops might be duplicated to have shorter buffer lifetimes. If
rematerialization occurs, it might be beneficial to investigate ways to reduce
memory requirements on the model side (e.g., using smaller batch sizes).
## Buffer Assignment
Immediately before we lower to LLVM IR, we run the buffer assignment passes that
will assign buffer slices to each instruction in the HLO graph. The buffer
assignment runs in several steps:
1. `HloDataflowAnalysis` assigns `HloValues` (essentially logical buffers) to
instructions. For in-place ops, the `HloValue` of an operand can be reused. An
op may define more than one `HloValue` (e.g. with a tuple result shape).
2. `HloAliasAnalysis` attempts to combine buffers for aliasing operations, and
computes a mapping from `HloValue` to `HloBuffer`.
3. `BufferAssignment` computes a mapping of `HloBuffers` to buffer slices inside
a big buffer in such a way that the same buffer slice is not used for different
`HloBuffers` with overlapping life times. For ops that may alias, it is ok that
there is a slight overlap (the end time of the one `HloBuffer` may coincide with
the start time of the other `HloBuffer`). When using the flag `-xla_dump_to`,
some information about buffer assignment is dumped to a file with the name
suffix "after_optimizations-buffer-assignment.txt".
## Thunks
After an HLO graph is optimized and scheduled, it is lowered into a
linear sequence of thunks for a specific backend (CPU or GPU).
In XLA, a **Thunk** is an abstraction of a self-contained unit of work that the
runtime executes. It might be a compiled kernel launch, specific operation,
library call, control-flow construct, collective communication, and so on. A
**Thunk Sequence** represents the entire executable for a specific backend.
### Thunk Emission
The process of converting a scheduled HLO computation into a thunk sequence is
called "thunk emission". This is handled by a dedicated emitter class in each
backend.
For the GPU Backend, this is handled by
[IrEmitterUnnested](https://github.com/openxla/xla/tree/main/xla/service/gpu/ir_emitter_unnested.h).
`EmitHloComputation` iterates through the scheduled list of HLO Instructions in
a computation and dispatches to a specialized `Emit...` method (e.g.,
`EmitFusion`, `EmitConvolutionThunk`, `EmitWhile`). Each of these methods
constructs the appropriate Thunk object(s) and appends them to the
thunk sequence.
For the CPU Backend,
[ThunkEmitter](https://github.com/openxla/xla/tree/main/xla/service/cpu/thunk_emitter.h)
performs this role and is organized in a similar manner. Final `ThunkSequence`
is embedded in the `CpuExecutable`.
Note that each instruction in the entry computation of an HLO module might
correspond to no (`kTuple`, `kConstant`, ..), one, or multiple (for example
sort instruction) thunks in the final thunk sequence.
### Command Buffers: Optimizing Execution on the GPU
Modern GPU hardware allows recording a sequence of GPU operations (kernel
launches, memory copies, etc.) once and then replaying the sequence multiple
times with minimal CPU overhead. This is a critical performance optimization,
especially for workloads with many small, fast-launching kernels. XLA uses
**Command Buffer** as an abstraction of CUDA Graphs or HIP Graphs. The core
interface is defined in
[GpuCommandBuffer](https://github.com/openxla/xla/tree/main/xla/stream_executor/gpu/gpu_command_buffer.h).
A command buffer is represented in a thunk sequence by
[CommandBufferThunk](https://github.com/openxla/xla/tree/main/xla/backends/gpu/runtime/command_buffer_thunk.h).
The emitter does not produce this thunk directly from HLO instructions. Instead,
this is done by
[CommandBufferConversionPass](https://github.com/openxla/xla/tree/main/xla/backends/gpu/runtime/command_buffer_conversion_pass.h)
that runs on the ThunkSequence itself.
The pass identifies contiguous sub-sequences of compatible thunks (e.g., a
series of `KernelThunk`s and `GemmThunk`s). It then replaces the found
sub-sequence with a single `CommandBufferThunk`. The new thunk encapsulates the
logic of the original thunks as a list of lightweight Command objects. When a
`CommandBufferThunk` executes for the first time on a given GPU stream, it
"records" its sequence of commands into a hardware command buffer. On all
subsequent executions, it simply issues a single command to the GPU to "replay"
the recorded sequence. This avoids the CPU overhead of launching each individual
kernel.
## Executable
The final product of the XLA compilation pipeline is a self-contained,
platform-specific
[Executable](https://github.com/openxla/xla/tree/main/xla/service/executable.h).
This object encapsulates all the information needed to run the compiled program
on a target device, such as a CPU or GPU. It is the bridge between the compiler
and the runtime. Modern runtimes like PJRT use slightly higher-level
abstractions (see
[PjRtExecutable](https://github.com/openxla/xla/tree/main/xla/pjrt/pjrt_executable.h)),
but these ultimately wrap a backend-specific executable.
An `Executable` contains several key pieces of information generated
during compilation. While the exact contents vary by backend, they generally
include:
- Compiled Code: This is the low-level machine code that will run on the device.
For CPUs, this is typically one or more object files. For GPUs, this is the
compiled device code in PTX or HSACO format, which is loaded onto the GPU at
runtime.
- Execution Plan (ThunkSequence): The core of the runtime logic. This is a
linear sequence of Thunk objects. Each thunk represents a single unit of work,
such as launching a kernel, calling a library function (e.g., cuBLAS), or
handling control flow. The runtime executes the program by iterating through
this sequence.
- Memory Layout (BufferAssignment): This critical piece of metadata, produced by
the BufferAssigner, describes the complete memory layout for the computation.
It specifies the size of every buffer and how memory is allocated and reused
for parameters, outputs, and temporary values. The runtime uses this to
allocate device memory and pass the correct pointers to each thunk.
- (optional) HLO Module: For debugging and profiling, the executable often
retains a reference to the final, optimized HloModule that it was compiled
from.
The creation of the final executable is orchestrated by the compiler for each
specific backend. The `RunBackend` method of a Compiler implementation is the
final step in the compilation process, which packages all the compiled artifacts
into an Executable object.
[GpuCompiler](https://github.com/openxla/xla/tree/main/xla/service/gpu/gpu_compiler.cc)
and
[CpuCompiler](https://github.com/openxla/xla/tree/main/xla/service/cpu/cpu_compiler.cc)
target GPU and CPU respectively.
When a user calls `Execute...` on an executable, the runtime uses the
`BufferAssignment` to allocate memory, and then invokes the `ThunkSequence` to
launch the operations on the device using the compiled code.
third_party/xla/docs/images/annotated_module.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 113 KiB

third_party/xla/docs/images/batch_group_counts.svg
Vendored
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 122 KiB

third_party/xla/docs/images/comaintainers.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 66 KiB

third_party/xla/docs/images/emitters-codegen-types.jpg
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 18 KiB

third_party/xla/docs/images/emitters-gelu.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 82 KiB

third_party/xla/docs/images/emitters-hero-types.jpg
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 20 KiB

third_party/xla/docs/images/emitters-partitioning.jpg
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 20 KiB

third_party/xla/docs/images/emitters-pipeline-overview.jpg
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 43 KiB

third_party/xla/docs/images/emitters-transpose.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 86 KiB

third_party/xla/docs/images/fused_module.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 308 KiB

third_party/xla/docs/images/gpu_compiler.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 52 KiB

third_party/xla/docs/images/gpu_pipeline.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 511 KiB

third_party/xla/docs/images/hlo_to_thunks.mermaid.txt
Vendored
+37
View File
@@ -0,0 +1,37 @@
flowchart TD
InitialHlo("HLO before optimizations")
OptimizedSheduledHlo("Optimized and scheduled HLO")
BufferAssignment("BufferAssignment")
ThunkSequence("ThunkSequence")
LlvmIr("LLVM IR")
MachineCode("Machine Code<br/><i>PTX, obj, ..</i>")
Executable("Executable")
%% Stages (Processes - Subroutine Shape)
HloPasses[[HLO Optimization]]
Scheduling[[Scheduling]]
BufferAssigner[[Buffer Assignment]]
ThunkEmitter[[Thunk Emission]]
LlvmCompiler[[LLVM Compilation]]
ExecutableBuilder[[Build Executable]]
%% Edges (Flow of Artifacts through Stages)
InitialHlo --> HloPasses
HloPasses --> Scheduling
Scheduling --> OptimizedSheduledHlo
OptimizedSheduledHlo --> BufferAssigner
BufferAssigner --> BufferAssignment
OptimizedSheduledHlo --> ThunkEmitter
BufferAssignment --> ThunkEmitter
ThunkEmitter --> ThunkSequence
ThunkEmitter -->
LlvmIr --> LlvmCompiler
LlvmCompiler --> MachineCode
MachineCode --> ExecutableBuilder
ThunkSequence --> ExecutableBuilder
BufferAssignment --> ExecutableBuilder
ExecutableBuilder --> Executable
third_party/xla/docs/images/hlo_to_thunks.svg
Vendored
+3
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 22 KiB

third_party/xla/docs/images/how-does-xla-work.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 17 KiB

third_party/xla/docs/images/indexing_analysis_softmax.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 147 KiB

third_party/xla/docs/images/indexing_analysis_transposes.svg
Vendored
+316
View File
@@ -0,0 +1,316 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet href="
data:text/css,
@import url(https://fonts.googleapis.com/css?family=Roboto:400,700);
svg text {
font-family: 'Roboto';
font-size: 12px;
}
%23node7:hover ~ %23edge8 text { fill: %231976d2; }
%23node7:hover ~ %23edge8 path { stroke: %231976d2; stroke-width: .2em; }
%23node7:hover ~ %23edge8 polygon { fill: %231976d2; stroke: %231976d2; stroke-width: .2em; }
%23node8:hover ~ %23edge8 text { fill: %23d32f2f; }
%23node8:hover ~ %23edge8 path { stroke: %23d32f2f; stroke-width: .2em; }
%23node8:hover ~ %23edge8 polygon { fill: %23d32f2f; stroke: %23d32f2f; stroke-width: .2em; }
%23node4:hover ~ %23edge7 text { fill: %231976d2; }
%23node4:hover ~ %23edge7 path { stroke: %231976d2; stroke-width: .2em; }
%23node4:hover ~ %23edge7 polygon { fill: %231976d2; stroke: %231976d2; stroke-width: .2em; }
%23node8:hover ~ %23edge7 text { fill: %23d32f2f; }
%23node8:hover ~ %23edge7 path { stroke: %23d32f2f; stroke-width: .2em; }
%23node8:hover ~ %23edge7 polygon { fill: %23d32f2f; stroke: %23d32f2f; stroke-width: .2em; }
%23node1:hover ~ %23edge4 text { fill: %231976d2; }
%23node1:hover ~ %23edge4 path { stroke: %231976d2; stroke-width: .2em; }
%23node1:hover ~ %23edge4 polygon { fill: %231976d2; stroke: %231976d2; stroke-width: .2em; }
%23node5:hover ~ %23edge4 text { fill: %23d32f2f; }
%23node5:hover ~ %23edge4 path { stroke: %23d32f2f; stroke-width: .2em; }
%23node5:hover ~ %23edge4 polygon { fill: %23d32f2f; stroke: %23d32f2f; stroke-width: .2em; }
%23node3:hover ~ %23edge3 text { fill: %231976d2; }
%23node3:hover ~ %23edge3 path { stroke: %231976d2; stroke-width: .2em; }
%23node3:hover ~ %23edge3 polygon { fill: %231976d2; stroke: %231976d2; stroke-width: .2em; }
%23node4:hover ~ %23edge3 text { fill: %23d32f2f; }
%23node4:hover ~ %23edge3 path { stroke: %23d32f2f; stroke-width: .2em; }
%23node4:hover ~ %23edge3 polygon { fill: %23d32f2f; stroke: %23d32f2f; stroke-width: .2em; }
%23node5:hover ~ %23edge5 text { fill: %231976d2; }
%23node5:hover ~ %23edge5 path { stroke: %231976d2; stroke-width: .2em; }
%23node5:hover ~ %23edge5 polygon { fill: %231976d2; stroke: %231976d2; stroke-width: .2em; }
%23node6:hover ~ %23edge5 text { fill: %23d32f2f; }
%23node6:hover ~ %23edge5 path { stroke: %23d32f2f; stroke-width: .2em; }
%23node6:hover ~ %23edge5 polygon { fill: %23d32f2f; stroke: %23d32f2f; stroke-width: .2em; }
%23node6:hover ~ %23edge6 text { fill: %231976d2; }
%23node6:hover ~ %23edge6 path { stroke: %231976d2; stroke-width: .2em; }
%23node6:hover ~ %23edge6 polygon { fill: %231976d2; stroke: %231976d2; stroke-width: .2em; }
%23node7:hover ~ %23edge6 text { fill: %23d32f2f; }
%23node7:hover ~ %23edge6 path { stroke: %23d32f2f; stroke-width: .2em; }
%23node7:hover ~ %23edge6 polygon { fill: %23d32f2f; stroke: %23d32f2f; stroke-width: .2em; }
%23node8:hover ~ %23edge9 text { fill: %231976d2; }
%23node8:hover ~ %23edge9 path { stroke: %231976d2; stroke-width: .2em; }
%23node8:hover ~ %23edge9 polygon { fill: %231976d2; stroke: %231976d2; stroke-width: .2em; }
%23node9:hover ~ %23edge9 text { fill: %23d32f2f; }
%23node9:hover ~ %23edge9 path { stroke: %23d32f2f; stroke-width: .2em; }
%23node9:hover ~ %23edge9 polygon { fill: %23d32f2f; stroke: %23d32f2f; stroke-width: .2em; }
%23node1:hover ~ %23edge1 text { fill: %231976d2; }
%23node1:hover ~ %23edge1 path { stroke: %231976d2; stroke-width: .2em; }
%23node1:hover ~ %23edge1 polygon { fill: %231976d2; stroke: %231976d2; stroke-width: .2em; }
%23node2:hover ~ %23edge1 text { fill: %23d32f2f; }
%23node2:hover ~ %23edge1 path { stroke: %23d32f2f; stroke-width: .2em; }
%23node2:hover ~ %23edge1 polygon { fill: %23d32f2f; stroke: %23d32f2f; stroke-width: .2em; }
%23node2:hover ~ %23edge2 text { fill: %231976d2; }
%23node2:hover ~ %23edge2 path { stroke: %231976d2; stroke-width: .2em; }
%23node2:hover ~ %23edge2 polygon { fill: %231976d2; stroke: %231976d2; stroke-width: .2em; }
%23node3:hover ~ %23edge2 text { fill: %23d32f2f; }
%23node3:hover ~ %23edge2 path { stroke: %23d32f2f; stroke-width: .2em; }
%23node3:hover ~ %23edge2 polygon { fill: %23d32f2f; stroke: %23d32f2f; stroke-width: .2em; }
" type="text/css"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<!-- Generated by graphviz version 2.44.1 (20201121.0304)
-->
<!-- Title: G Pages: 1 -->
<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" width="290pt" height="549pt" viewBox="0.00 0.00 290.25 548.60"><script>
(function() {
var enablePanZoom = false;
var enablePanZoomControls = false;
var panZoomScriptUrl = "https://graphviz.corp.google.com/svg-pan-zoom.js";
var isEmbedded = true;
try {
// This condition is true when the SVG is being displayed in a
// top-level browser window. If the parent location's href is
// different, or we can't access it (due to security constraints
// on accessing javascript properties on a different domain),
// then the document is embedded.
if (window.location.href == window.parent.location.href) {
isEmbedded = false;
}
} catch (e) {}
if (!isEmbedded) enablePanZoom = true;
if (enablePanZoom &amp;&amp; typeof svgPanZoom === 'undefined') {
var panZoomScript =
document.createElementNS('http://www.w3.org/2000/svg', 'script');
panZoomScript.setAttributeNS(
'http://www.w3.org/1999/xlink', 'xlink:href', panZoomScriptUrl);
document.currentScript.parentElement.appendChild(panZoomScript);
}
window.onload = function() {
var svg = document.getElementsByTagName("svg")[0];
if (!isEmbedded) {
svg.removeAttribute("width");
svg.removeAttribute("height");
}
if (enablePanZoom) {
var altPressed = false;
svgPanZoom(svg, {
minZoom: 0.2,
maxZoom: 100,
controlIconsEnabled: enablePanZoomControls,
preventMouseEventsDefault: isEmbedded,
beforeZoom: function(oldZoom, newZoom) { return !altPressed; },
beforePan: function(oldPan, newPan) { return !altPressed; },
});
document.onkeydown = function(e) {
altPressed = e.altKey;
};
document.onkeyup = function(e) {
altPressed = e.altKey;
};
}
var links = document.getElementsByTagName("a");
for (var i = 0; i &lt; links.length; i++) {
if (!links[i].getAttribute("target")) {
links[i].setAttribute("target", "_top");
}
}
};
})();
</script>
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 544.6)">
<title>G</title>
<g id="a_graph0"><a xlink:title=" ">
<polygon fill="white" stroke="none" points="-4,4 -4,-544.6 286.25,-544.6 286.25,4 -4,4"/>
<text text-anchor="start" x="14" y="-510.3" font-family="Times,serif" font-weight="bold" font-size="14.00">Computation f (in fusion instruction fusion)</text>
</a>
</g>
<!-- 5734836738048 -->
<g id="node1" class="node">
<title>5734836738048</title>
<polygon fill="#ffb74d" stroke="#c88719" points="207.25,-504.6 75,-504.6 75,-468.6 207.25,-468.6 207.25,-504.6"/>
<text text-anchor="start" x="104.75" y="-488.3" font-family="Times,serif" font-weight="bold" font-size="14.00">Parameter 0</text>
<text text-anchor="start" x="83" y="-474.3" font-family="Times,serif" font-size="14.00">f32[20,10,50]{2,1,0}</text>
</g>
<!-- 5734836737024 -->
<g id="node2" class="node">
<title>5734836737024</title>
<polygon fill="#c8e6c9" stroke="#97b498" points="132.25,-432.6 0,-432.6 0,-368.6 132.25,-368.6 132.25,-432.6"/>
<text text-anchor="start" x="38.38" y="-416.3" font-family="Times,serif" font-weight="bold" font-size="14.00">transpose</text>
<text text-anchor="start" x="22.62" y="-402.3" font-family="Times,serif" font-size="14.00">lhs_transpose_1</text>
<text text-anchor="start" x="10.25" y="-388.3" font-family="Times,serif" font-size="14.00">dimensions={1,0,2}</text>
<text text-anchor="start" x="8" y="-374.3" font-family="Times,serif" font-size="14.00">f32[10,20,50]{2,1,0}</text>
</g>
<!-- 5734836738048&#45;&gt;5734836737024 -->
<g id="edge1" class="edge">
<title>5734836738048-&gt;5734836737024</title>
<g id="a_edge1"><a xlink:title="p0 -&gt; lhs_transpose_1">
<path fill="none" stroke="black" d="M125.58,-468.19C118.62,-460.39 110.1,-450.85 101.7,-441.44"/>
<polygon fill="black" stroke="black" points="104.35,-439.16 95.08,-434.03 99.13,-443.82 104.35,-439.16"/>
</a>
</g>
</g>
<!-- 5734836733952 -->
<g id="node5" class="node">
<title>5734836733952</title>
<polygon fill="#c8e6c9" stroke="#97b498" points="282.25,-432.6 150,-432.6 150,-368.6 282.25,-368.6 282.25,-432.6"/>
<text text-anchor="start" x="188.38" y="-416.3" font-family="Times,serif" font-weight="bold" font-size="14.00">transpose</text>
<text text-anchor="start" x="172.25" y="-402.3" font-family="Times,serif" font-size="14.00">rhs_transpose_1</text>
<text text-anchor="start" x="160.25" y="-388.3" font-family="Times,serif" font-size="14.00">dimensions={2,1,0}</text>
<text text-anchor="start" x="158" y="-374.3" font-family="Times,serif" font-size="14.00">f32[50,10,20]{2,1,0}</text>
</g>
<!-- 5734836738048&#45;&gt;5734836733952 -->
<g id="edge4" class="edge">
<title>5734836738048-&gt;5734836733952</title>
<g id="a_edge4"><a xlink:title="p0 -&gt; rhs_transpose_1">
<path fill="none" stroke="black" d="M156.67,-468.19C163.63,-460.39 172.15,-450.85 180.55,-441.44"/>
<polygon fill="black" stroke="black" points="183.12,-443.82 187.17,-434.03 177.9,-439.16 183.12,-443.82"/>
</a>
</g>
</g>
<!-- 5734836736000 -->
<g id="node3" class="node">
<title>5734836736000</title>
<polygon fill="white" stroke="#9e9e9e" points="132.25,-332.6 0,-332.6 0,-282.6 132.25,-282.6 132.25,-332.6"/>
<text text-anchor="start" x="32.75" y="-316.3" font-family="Times,serif" font-weight="bold" font-size="14.00">exponential</text>
<text text-anchor="start" x="51.88" y="-302.3" font-family="Times,serif" font-size="14.00">lhs_e</text>
<text text-anchor="start" x="8" y="-288.3" font-family="Times,serif" font-size="14.00">f32[10,20,50]{2,1,0}</text>
</g>
<!-- 5734836737024&#45;&gt;5734836736000 -->
<g id="edge2" class="edge">
<title>5734836737024-&gt;5734836736000</title>
<g id="a_edge2"><a xlink:title="lhs_transpose_1 -&gt; lhs_e">
<path fill="none" stroke="black" d="M66.12,-368.28C66.12,-360.64 66.12,-352.39 66.12,-344.55"/>
<polygon fill="black" stroke="black" points="69.63,-344.59 66.13,-334.59 62.63,-344.59 69.63,-344.59"/>
</a>
</g>
</g>
<!-- 5734836734976 -->
<g id="node4" class="node">
<title>5734836734976</title>
<polygon fill="#c8e6c9" stroke="#97b498" points="132.25,-246.6 0,-246.6 0,-182.6 132.25,-182.6 132.25,-246.6"/>
<text text-anchor="start" x="38.38" y="-230.3" font-family="Times,serif" font-weight="bold" font-size="14.00">transpose</text>
<text text-anchor="start" x="22.62" y="-216.3" font-family="Times,serif" font-size="14.00">lhs_transpose_2</text>
<text text-anchor="start" x="10.25" y="-202.3" font-family="Times,serif" font-size="14.00">dimensions={0,2,1}</text>
<text text-anchor="start" x="8" y="-188.3" font-family="Times,serif" font-size="14.00">f32[10,50,20]{2,1,0}</text>
</g>
<!-- 5734836736000&#45;&gt;5734836734976 -->
<g id="edge3" class="edge">
<title>5734836736000-&gt;5734836734976</title>
<g id="a_edge3"><a xlink:title="lhs_e -&gt; lhs_transpose_2">
<path fill="none" stroke="black" d="M66.12,-282.21C66.12,-274.84 66.12,-266.52 66.12,-258.32"/>
<polygon fill="black" stroke="black" points="69.63,-258.43 66.13,-248.43 62.63,-258.43 69.63,-258.43"/>
</a>
</g>
</g>
<!-- 5734836698112 -->
<g id="node8" class="node">
<title>5734836698112</title>
<polygon fill="white" stroke="#9e9e9e" points="207.25,-146.6 75,-146.6 75,-110.6 207.25,-110.6 207.25,-146.6"/>
<text text-anchor="start" x="130.25" y="-130.3" font-family="Times,serif" font-weight="bold" font-size="14.00">add</text>
<text text-anchor="start" x="83" y="-116.3" font-family="Times,serif" font-size="14.00">f32[10,50,20]{2,1,0}</text>
</g>
<!-- 5734836734976&#45;&gt;5734836698112 -->
<g id="edge7" class="edge">
<title>5734836734976-&gt;5734836698112</title>
<g id="a_edge7"><a xlink:title="lhs_transpose_2 -&gt; add">
<path fill="none" stroke="black" d="M94.05,-182.32C101.97,-173.46 110.49,-163.91 118.08,-155.41"/>
<polygon fill="black" stroke="black" points="120.53,-157.92 124.58,-148.13 115.31,-153.26 120.53,-157.92"/>
</a>
</g>
<text text-anchor="middle" x="119.82" y="-161.86" font-family="Times,serif" font-size="14.00">0</text>
</g>
<!-- 5734836732928 -->
<g id="node6" class="node">
<title>5734836732928</title>
<polygon fill="white" stroke="#9e9e9e" points="282.25,-332.6 150,-332.6 150,-282.6 282.25,-282.6 282.25,-332.6"/>
<text text-anchor="start" x="182.75" y="-316.3" font-family="Times,serif" font-weight="bold" font-size="14.00">exponential</text>
<text text-anchor="start" x="195.88" y="-302.3" font-family="Times,serif" font-size="14.00">rhs_log</text>
<text text-anchor="start" x="158" y="-288.3" font-family="Times,serif" font-size="14.00">f32[50,10,20]{2,1,0}</text>
</g>
<!-- 5734836733952&#45;&gt;5734836732928 -->
<g id="edge5" class="edge">
<title>5734836733952-&gt;5734836732928</title>
<g id="a_edge5"><a xlink:title="rhs_transpose_1 -&gt; rhs_log">
<path fill="none" stroke="black" d="M216.12,-368.28C216.12,-360.64 216.12,-352.39 216.12,-344.55"/>
<polygon fill="black" stroke="black" points="219.63,-344.59 216.13,-334.59 212.63,-344.59 219.63,-344.59"/>
</a>
</g>
</g>
<!-- 5734836731904 -->
<g id="node7" class="node">
<title>5734836731904</title>
<polygon fill="#c8e6c9" stroke="#97b498" points="282.25,-246.6 150,-246.6 150,-182.6 282.25,-182.6 282.25,-246.6"/>
<text text-anchor="start" x="188.38" y="-230.3" font-family="Times,serif" font-weight="bold" font-size="14.00">transpose</text>
<text text-anchor="start" x="172.25" y="-216.3" font-family="Times,serif" font-size="14.00">rhs_transpose_2</text>
<text text-anchor="start" x="160.25" y="-202.3" font-family="Times,serif" font-size="14.00">dimensions={1,0,2}</text>
<text text-anchor="start" x="158" y="-188.3" font-family="Times,serif" font-size="14.00">f32[10,50,20]{2,1,0}</text>
</g>
<!-- 5734836732928&#45;&gt;5734836731904 -->
<g id="edge6" class="edge">
<title>5734836732928-&gt;5734836731904</title>
<g id="a_edge6"><a xlink:title="rhs_log -&gt; rhs_transpose_2">
<path fill="none" stroke="black" d="M216.12,-282.21C216.12,-274.84 216.12,-266.52 216.12,-258.32"/>
<polygon fill="black" stroke="black" points="219.63,-258.43 216.13,-248.43 212.63,-258.43 219.63,-258.43"/>
</a>
</g>
</g>
<!-- 5734836731904&#45;&gt;5734836698112 -->
<g id="edge8" class="edge">
<title>5734836731904-&gt;5734836698112</title>
<g id="a_edge8"><a xlink:title="rhs_transpose_2 -&gt; add">
<path fill="none" stroke="black" d="M188.2,-182.32C180.28,-173.46 171.76,-163.91 164.17,-155.41"/>
<polygon fill="black" stroke="black" points="166.94,-153.26 157.67,-148.13 161.72,-157.92 166.94,-153.26"/>
</a>
</g>
<text text-anchor="middle" x="175.04" y="-150.6" font-family="Times,serif" font-size="14.00">1</text>
</g>
<!-- cluster_5734844288888 -->
<g id="node9" class="node">
<title>cluster_5734844288888</title>
<g id="a_node9"><a xlink:title=" ">
<ellipse fill="#bcaaa4" stroke="#8c7b75" cx="141.12" cy="-37.3" rx="37.3" ry="37.3"/>
<text text-anchor="start" x="122.75" y="-33" font-family="Times,serif" font-size="14.00">ROOT</text>
</a>
</g>
</g>
<!-- 5734836698112&#45;&gt;cluster_5734844288888 -->
<g id="edge9" class="edge">
<title>5734836698112-&gt;cluster_5734844288888</title>
<g id="a_edge9"><a xlink:title=" ">
<path fill="none" stroke="black" d="M141.12,-110.38C141.12,-103.4 141.12,-94.97 141.12,-86.36"/>
<polygon fill="black" stroke="black" points="144.63,-86.57 141.13,-76.57 137.63,-86.57 144.63,-86.57"/>
</a>
</g>
</g>
</g>
</svg>
third_party/xla/docs/images/layout_assigned_module.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 110 KiB

third_party/xla/docs/images/lowered_hlo.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 203 KiB

third_party/xla/docs/images/oom_debugging_example_memory_viewer.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 150 KiB

third_party/xla/docs/images/openxla.svg
Vendored
+728
View File
@@ -0,0 +1,728 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<svg
id="Layer_1"
data-name="Layer 1"
viewBox="0 0 857.25321 368.17926"
version="1.1"
sodipodi:docname="openxla.svg"
width="780"
height="335"
inkscape:version="1.4 (e7c3feb1, 2024-10-09)"
xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns="http://www.w3.org/2000/svg"
xmlns:svg="http://www.w3.org/2000/svg">
<sodipodi:namedview
id="namedview91"
pagecolor="#747b82"
bordercolor="#000000"
borderopacity="0.25"
inkscape:showpageshadow="2"
inkscape:pageopacity="0.0"
inkscape:pagecheckerboard="0"
inkscape:deskcolor="#d1d1d1"
showgrid="false"
inkscape:zoom="0.5718075"
inkscape:cx="201.99106"
inkscape:cy="229.09808"
inkscape:window-width="1656"
inkscape:window-height="747"
inkscape:window-x="0"
inkscape:window-y="38"
inkscape:window-maximized="0"
inkscape:current-layer="Layer_1" />
<defs
id="defs2">
<style
id="style1">
.cls-1 {
fill: #ea80fc;
}
.cls-1, .cls-2, .cls-3, .cls-4, .cls-5, .cls-6, .cls-7, .cls-8, .cls-9 {
stroke: #dce0df;
stroke-linejoin: round;
stroke-width: .13px;
}
.cls-2 {
fill: #3367d6;
}
.cls-3 {
fill: #4d81f3;
}
.cls-4 {
fill: #26a69a;
}
.cls-5 {
fill: #2a56c6;
}
.cls-6 {
fill: #9c27b0;
}
.cls-7 {
fill: #6a1b9a;
}
.cls-8 {
fill: #00695c;
}
.cls-9 {
fill: #00796b;
}
.cls-10 {
clip-path: url(#clippath);
}
.cls-11, .cls-12, .cls-13, .cls-14, .cls-15, .cls-16, .cls-17, .cls-18 {
stroke-width: 0px;
}
.cls-11, .cls-19, .cls-20, .cls-21 {
fill: none;
}
<!--
darkblue = #1a344c
midblue = #458ae5
lightblue = #6fb6e0
-->
.openxla-box {
fill: #fff!important;
filter: drop-shadow( 2px 2px 3px rgba(26, 52, 76, .5));
}
.cls-12 {
fill: url(#linear-gradient);
}
.cls-19 {
stroke-width: .65px;
}
.cls-19, .cls-20, .cls-21 {
stroke: #1a344c;
stroke-miterlimit: 10;
}
.cls-20 {
stroke-width: .65px;
}
.cls-21 {
stroke-width: .54px;
}
.cls-22 {
clip-path: url(#clippath-1);
}
.cls-13 {
fill: url(#linear-gradient-2);
}
.cls-14 {
fill: #458ae5;
}
.cls-15 {
fill: #6fb6e0;
}
.cls-16 {
fill: #fff;
}
.cls-17 {
fill: #1a344c;
}
.cls-18 {
fill: #ee4c2c;
}
</style>
<clipPath
id="clippath">
<polygon
class="cls-11"
points="237.92,228.11 237.92,222.82 227.74,217.03 227.74,240.83 231.81,238.47 231.81,231.78 234.89,233.54 234.84,228.97 231.81,227.21 231.81,224.54 "
id="polygon1" />
</clipPath>
<linearGradient
id="linear-gradient"
x1="-348.29001"
y1="-1465.28"
x2="-323.32001"
y2="-1465.28"
gradientTransform="matrix(1,0,0,-1,564,-1236.37)"
gradientUnits="userSpaceOnUse">
<stop
offset="0"
stop-color="#ff6f00"
id="stop1" />
<stop
offset="1"
stop-color="#ffa800"
id="stop2" />
</linearGradient>
<clipPath
id="clippath-1">
<polygon
class="cls-11"
points="226.83,217.03 226.83,240.83 222.76,238.47 222.76,224.54 216.66,228.11 216.66,222.82 "
id="polygon2" />
</clipPath>
<linearGradient
id="linear-gradient-2"
x1="-348.42999"
x2="-323.45999"
xlink:href="#linear-gradient" />
<linearGradient
inkscape:collect="always"
xlink:href="#linear-gradient"
id="linearGradient91"
gradientUnits="userSpaceOnUse"
gradientTransform="matrix(1,0,0,-1,564,-1236.37)"
x1="-348.29001"
y1="-1465.28"
x2="-323.32001"
y2="-1465.28" />
</defs>
<rect
class="cls-20"
x="36.576595"
y="57.619633"
width="194.16"
height="62.799999"
rx="11.07"
ry="11.07"
id="rect2" />
<g
id="Layer_2"
data-name="Layer 2"
transform="translate(-147.3734,-139.91037)">
<g
id="Layer_1-2"
data-name="Layer 1-2">
<polygon
class="cls-3"
points="263.67,324 257.34,324 254.17,329.48 260.5,329.48 "
id="polygon3" />
<polygon
class="cls-3"
points="257.34,334.96 251.01,334.96 254.17,329.48 260.5,329.48 "
id="polygon4" />
<polygon
class="cls-3"
points="263.67,334.96 266.83,329.48 260.5,329.48 257.34,334.96 "
id="polygon5" />
<polygon
class="cls-3"
points="269.99,334.96 273.16,329.48 266.83,329.48 263.67,334.96 "
id="polygon6" />
<polygon
class="cls-3"
points="276.32,324 269.99,324 266.83,329.48 273.16,329.48 "
id="polygon7" />
<polygon
class="cls-3"
points="279.49,318.52 273.16,318.52 269.99,324 276.32,324 "
id="polygon8" />
<polygon
class="cls-3"
points="282.65,313.04 276.32,313.04 273.16,318.52 279.49,318.52 "
id="polygon9" />
<polygon
class="cls-3"
points="285.82,307.56 279.49,307.56 276.32,313.04 282.65,313.04 "
id="polygon10" />
<polygon
class="cls-5"
points="257.34,334.96 251.01,334.96 254.17,340.44 260.5,340.44 "
id="polygon11" />
<polygon
class="cls-5"
points="263.67,334.96 266.83,340.44 260.5,340.44 257.34,334.96 "
id="polygon12" />
<polygon
class="cls-5"
points="269.99,334.96 273.16,340.44 266.83,340.44 263.67,334.96 "
id="polygon13" />
<polygon
class="cls-9"
points="273.16,340.44 276.32,334.96 273.16,329.48 269.99,334.96 "
id="polygon14" />
<polygon
class="cls-9"
points="276.32,324 282.65,324 279.49,318.52 "
id="polygon15" />
<polygon
class="cls-9"
points="285.82,318.52 282.65,313.04 279.49,318.52 282.65,324 "
id="polygon16" />
<polygon
class="cls-2"
points="260.5,329.48 266.83,329.48 263.67,324 "
id="polygon17" />
<polygon
class="cls-4"
points="279.49,329.48 282.65,324 276.32,324 273.16,329.48 "
id="polygon18" />
<polygon
class="cls-4"
points="285.82,329.48 288.98,324 282.65,324 279.49,329.48 "
id="polygon19" />
<polygon
class="cls-6"
points="292.15,318.52 295.31,313.04 292.15,307.56 288.98,313.04 "
id="polygon20" />
<polygon
class="cls-6"
points="295.31,324 298.48,318.52 295.31,313.04 292.15,318.52 "
id="polygon21" />
<polygon
class="cls-6"
points="298.48,329.48 301.64,324 298.48,318.52 295.31,324 "
id="polygon22" />
<polygon
class="cls-6"
points="301.64,334.96 304.81,329.48 301.64,324 298.48,329.48 "
id="polygon23" />
<polygon
class="cls-6"
points="304.81,340.44 307.97,334.96 304.81,329.48 301.64,334.96 "
id="polygon24" />
<polygon
class="cls-6"
points="307.97,313.04 304.81,307.56 301.64,313.04 304.81,318.52 "
id="polygon25" />
<polygon
class="cls-6"
points="301.64,324 298.48,318.52 301.64,313.04 304.81,318.52 "
id="polygon26" />
<polygon
class="cls-6"
points="298.48,329.48 295.31,324 292.15,329.48 295.31,334.96 "
id="polygon27" />
<polygon
class="cls-6"
points="295.31,334.96 292.15,340.44 288.98,334.96 292.15,329.48 "
id="polygon28" />
<polygon
class="cls-7"
points="292.15,340.44 285.82,340.44 282.65,334.96 288.98,334.96 "
id="polygon29" />
<polygon
class="cls-8"
points="282.65,334.96 279.49,329.48 273.16,329.48 276.32,334.96 "
id="polygon30" />
<polygon
class="cls-8"
points="282.65,334.96 285.82,329.48 279.49,329.48 "
id="polygon31" />
<polygon
class="cls-8"
points="292.15,318.52 285.82,318.52 288.98,324 295.31,324 "
id="polygon32" />
<polygon
class="cls-8"
points="292.15,318.52 288.98,313.04 282.65,313.04 285.82,318.52 "
id="polygon33" />
<polygon
class="cls-8"
points="298.48,340.44 304.81,340.44 301.64,334.96 295.31,334.96 "
id="polygon34" />
<polygon
class="cls-8"
points="301.64,334.96 298.48,329.48 295.31,334.96 "
id="polygon35" />
<polygon
class="cls-1"
points="288.98,313.04 292.15,307.56 285.82,307.56 282.65,313.04 "
id="polygon36" />
<polygon
class="cls-1"
points="288.98,334.96 292.15,329.48 285.82,329.48 282.65,334.96 "
id="polygon37" />
<polygon
class="cls-1"
points="292.15,329.48 295.31,324 288.98,324 285.82,329.48 "
id="polygon38" />
<polygon
class="cls-1"
points="301.64,313.04 304.81,307.56 298.48,307.56 295.31,313.04 "
id="polygon39" />
<polygon
class="cls-1"
points="301.64,313.04 298.48,318.52 295.31,313.04 "
id="polygon40" />
</g>
</g>
<g
id="g44"
transform="translate(-147.3734,-139.91037)">
<g
class="cls-10"
clip-path="url(#clippath)"
id="g40">
<path
class="cls-12"
d="m 215.71,216.94 h 24.97 v 23.93 h -24.97 z"
id="path40"
style="fill:url(#linearGradient91)" />
</g>
<g
class="cls-22"
clip-path="url(#clippath-1)"
id="g41">
<path
class="cls-13"
d="m 215.57,216.94 h 24.97 v 23.93 h -24.97 z"
id="path41"
style="fill:url(#linear-gradient-2)" />
</g>
<g
id="g43">
<path
class="cls-17 component-text"
d="m 256.33,224.41 h -4.52 v 12.53 h -2.53 v -12.53 h -4.52 v -2.04 h 11.58 v 2.04 z"
id="path42" />
<path
class="cls-17"
d="m 259.36,237.12 c -1.54,0 -2.8,-0.5 -3.75,-1.45 -0.95,-0.95 -1.45,-2.26 -1.45,-3.89 v -0.32 c 0,-1 0.18,-1.99 0.63,-2.9 0.41,-0.81 1,-1.49 1.76,-1.99 0.77,-0.5 1.63,-0.72 2.53,-0.72 1.49,0 2.62,0.45 3.44,1.4 0.82,0.95 1.22,2.26 1.22,3.98 v 1 h -7.1 c 0.05,0.81 0.36,1.54 0.9,2.13 0.54,0.54 1.22,0.81 1.99,0.77 1.09,0.05 2.08,-0.5 2.71,-1.36 l 1.31,1.27 c -0.45,0.63 -1.04,1.18 -1.72,1.49 -0.81,0.45 -1.63,0.63 -2.49,0.59 z m -0.27,-9.27 c -0.63,-0.05 -1.22,0.23 -1.63,0.68 -0.45,0.54 -0.72,1.22 -0.77,1.95 h 4.66 v -0.18 c -0.05,-0.81 -0.27,-1.45 -0.63,-1.85 -0.45,-0.36 -1.04,-0.63 -1.63,-0.59 z m 8.78,-1.76 0.09,1.27 c 0.77,-0.95 1.95,-1.49 3.17,-1.45 2.26,0 3.39,1.31 3.44,3.89 v 7.15 h -2.44 v -7.01 c 0,-0.68 -0.14,-1.18 -0.45,-1.54 -0.32,-0.32 -0.77,-0.5 -1.45,-0.5 -0.95,-0.05 -1.81,0.5 -2.22,1.31 v 7.69 h -2.44 v -10.86 c 0,0 2.31,0.05 2.31,0.05 z m 14.57,7.92 c 0,-0.41 -0.18,-0.77 -0.54,-1 -0.54,-0.32 -1.18,-0.5 -1.76,-0.59 -0.72,-0.14 -1.4,-0.36 -2.08,-0.68 -1.22,-0.59 -1.81,-1.45 -1.81,-2.53 0,-0.9 0.45,-1.81 1.18,-2.35 0.77,-0.63 1.81,-0.95 2.99,-0.95 1.31,0 2.35,0.32 3.12,0.95 0.77,0.59 1.22,1.54 1.18,2.49 h -2.44 c 0,-0.45 -0.18,-0.86 -0.54,-1.18 -0.41,-0.32 -0.86,-0.5 -1.4,-0.45 -0.45,0 -0.9,0.09 -1.31,0.36 -0.32,0.23 -0.5,0.59 -0.5,1 0,0.36 0.18,0.68 0.45,0.86 0.32,0.23 0.95,0.41 1.9,0.63 0.77,0.14 1.54,0.41 2.26,0.77 0.5,0.23 0.9,0.59 1.22,1.04 0.27,0.45 0.41,0.95 0.41,1.49 0,0.95 -0.45,1.81 -1.22,2.35 -0.81,0.59 -1.85,0.9 -3.17,0.9 -0.81,0 -1.63,-0.14 -2.35,-0.5 -0.63,-0.27 -1.22,-0.72 -1.63,-1.31 -0.36,-0.54 -0.59,-1.18 -0.59,-1.81 h 2.35 c 0,0.5 0.23,1 0.63,1.31 0.45,0.32 1.04,0.5 1.58,0.45 0.63,0 1.13,-0.14 1.45,-0.36 0.45,-0.18 0.63,-0.54 0.63,-0.9 v 0 z m 3.66,-2.58 c 0,-1 0.18,-1.99 0.63,-2.85 0.41,-0.81 1,-1.49 1.76,-1.95 0.81,-0.45 1.72,-0.72 2.62,-0.68 1.45,0 2.67,0.45 3.57,1.4 0.9,0.95 1.4,2.17 1.49,3.75 v 0.59 c 0,1 -0.18,1.95 -0.63,2.85 -0.36,0.81 -1,1.49 -1.76,1.95 -0.81,0.45 -1.72,0.72 -2.67,0.68 -1.54,0 -2.76,-0.5 -3.66,-1.54 -0.9,-1 -1.36,-2.35 -1.4,-4.07 l 0.05,-0.14 z m 2.4,0.23 c 0,1.13 0.23,1.99 0.68,2.62 0.81,1.04 2.31,1.27 3.39,0.45 0.18,-0.14 0.32,-0.27 0.45,-0.45 0.45,-0.63 0.68,-1.58 0.68,-2.8 0,-1.09 -0.23,-1.95 -0.72,-2.62 -0.77,-1.04 -2.26,-1.27 -3.35,-0.5 -0.18,0.14 -0.36,0.32 -0.5,0.45 -0.36,0.63 -0.63,1.58 -0.63,2.85 z m 14.97,-3.3 c -0.32,-0.05 -0.68,-0.09 -1,-0.09 -1.13,0 -1.85,0.41 -2.26,1.27 v 7.42 h -2.44 V 226.1 h 2.31 l 0.05,1.22 c 0.59,-0.95 1.4,-1.4 2.44,-1.4 0.27,0 0.59,0.05 0.86,0.14 l 0.05,2.31 v 0 z m 10.18,2.4 h -5.88 v 6.2 h -2.53 v -14.57 h 9.27 v 2.04 h -6.74 v 4.34 h 5.88 z m 4.84,6.2 h -2.44 v -14.61 h 2.44 z m 1.76,-5.52 c 0,-1 0.18,-1.99 0.63,-2.85 0.41,-0.81 1,-1.49 1.76,-1.95 0.81,-0.45 1.72,-0.72 2.62,-0.68 1.45,0 2.67,0.45 3.57,1.4 0.9,0.95 1.4,2.17 1.49,3.75 v 0.59 c 0,1 -0.18,1.95 -0.59,2.85 -0.36,0.81 -1,1.49 -1.76,1.95 -0.81,0.45 -1.72,0.72 -2.67,0.68 -1.54,0 -2.76,-0.5 -3.66,-1.54 -0.9,-1 -1.4,-2.35 -1.4,-4.07 v -0.14 0 z m 2.44,0.23 c 0,1.13 0.23,1.99 0.68,2.62 0.45,0.63 1.18,1 1.95,0.95 0.77,0.05 1.49,-0.32 1.9,-0.95 0.45,-0.63 0.68,-1.58 0.68,-2.8 0,-1.09 -0.23,-1.95 -0.72,-2.62 -0.77,-1.04 -2.26,-1.27 -3.35,-0.5 -0.18,0.14 -0.36,0.32 -0.5,0.45 -0.41,0.63 -0.63,1.58 -0.63,2.85 z m 18.64,1.95 1.72,-7.46 h 2.35 l -2.94,10.86 h -1.99 l -2.31,-7.46 -2.31,7.46 h -1.99 l -2.99,-10.86 h 2.4 l 1.76,7.42 2.22,-7.42 h 1.85 l 2.22,7.46 v 0 z"
id="path43" />
</g>
</g>
<rect
class="cls-20"
x="36.576595"
y="152.68964"
width="194.16"
height="62.799999"
rx="11.07"
ry="11.07"
id="rect44" />
<path
class="cls-20"
d="m 219.6566,310.55963 h -172.01 c -6.12,0 -11.07,-4.96 -11.07,-11.07 v -40.66 c 0,-6.12 4.96,-11.07 11.07,-11.07 h 172.01 c 6.12,0 11.07,4.96 11.07,11.07 v 40.66 c 0,6.12 -4.96,11.07 -11.07,11.07 z"
id="path44" />
<g
id="g53"
transform="translate(-147.3734,-139.91037)">
<g
id="g45">
<path
class="cls-18"
d="m 237.41,412.75 -1.59,1.59 c 2.6,2.6 2.6,6.79 0,9.34 -2.6,2.6 -6.79,2.6 -9.34,0 -2.6,-2.6 -2.6,-6.79 0,-9.34 v 0 l 4.12,-4.12 0.58,-0.58 v 0 -3.11 l -6.21,6.21 c -3.47,3.47 -3.47,9.05 0,12.52 3.47,3.47 9.05,3.47 12.45,0 3.47,-3.49 3.47,-9.05 0,-12.52 z"
id="path45" />
<circle
class="cls-18"
cx="234.3"
cy="411.20999"
r="1.16"
id="circle45" />
</g>
<g
id="g52">
<path
class="cls-17"
d="m 253.47,420.69 h -2.67 v 6.86 h -2.02 v -19.53 h 4.91 c 5.13,0 7.58,2.53 7.58,6.07 0.07,4.36 -2.96,6.6 -7.8,6.6 z m 0.22,-10.72 h -2.82 v 8.98 l 2.75,-0.07 c 3.68,-0.07 5.71,-1.52 5.71,-4.55 0,-2.84 -2.02,-4.36 -5.63,-4.36 z"
id="path46" />
<path
class="cls-17"
d="m 270.32,427.48 -1.16,3.11 c -1.3,3.47 -2.67,4.48 -4.62,4.48 -1.08,0 -1.88,-0.29 -2.75,-0.65 l 0.58,-1.81 c 0.65,0.36 1.37,0.65 2.17,0.65 1.08,0 1.88,-0.58 2.96,-3.32 l 0.94,-2.53 -5.56,-14.11 h 2.09 l 4.48,11.8 4.41,-11.8 h 2.02 z"
id="path47" />
<path
class="cls-17"
d="m 282.48,409.98 v 17.65 h -2.02 v -17.65 h -6.86 v -1.88 h 15.7 v 1.88 h -6.81 z"
id="path48" />
<path
class="cls-17"
d="m 295.01,427.99 c -3.97,0 -6.86,-2.96 -6.86,-7.51 0,-4.55 3.03,-7.58 7.08,-7.58 4.05,0 6.86,2.96 6.86,7.51 -0.07,4.55 -3.11,7.58 -7.08,7.58 z m 0.07,-13.29 c -3.03,0 -4.98,2.38 -4.98,5.78 0,3.4 2.02,5.85 5.06,5.85 3.04,0 4.98,-2.38 4.98,-5.78 0,-3.4 -2.02,-5.85 -5.06,-5.85 z"
id="path49" />
<path
class="cls-17"
d="m 306.92,427.63 h -1.95 V 413.3 l 1.95,-0.43 v 3.03 c 0.94,-1.81 2.31,-3.03 4.19,-3.03 0.94,0 1.81,0.29 2.53,0.65 l -0.51,1.81 c -0.65,-0.36 -1.37,-0.65 -2.17,-0.65 -1.52,0 -2.89,1.16 -4.05,3.68 v 9.27 0 z"
id="path50" />
<path
class="cls-17"
d="m 321.25,427.99 c -4.33,0 -7.01,-3.11 -7.01,-7.51 0,-4.4 2.96,-7.58 7.01,-7.58 1.73,0 3.25,0.43 4.48,1.23 l -0.51,1.73 c -1.08,-0.72 -2.46,-1.16 -3.97,-1.16 -3.11,0 -4.98,2.31 -4.98,5.71 0,3.4 2.02,5.78 5.06,5.78 1.44,0 2.89,-0.43 3.97,-1.16 l 0.43,1.81 c -1.3,0.72 -2.82,1.16 -4.48,1.16 z"
id="path51" />
<path
class="cls-17"
d="m 337.67,427.63 v -9.27 c 0,-2.53 -1.01,-3.61 -3.03,-3.61 -1.66,0 -3.25,0.87 -4.41,2.02 v 10.86 h -1.95 v -21.05 l 1.95,-0.43 v 9.05 c 1.52,-1.52 3.4,-2.24 4.98,-2.24 2.75,0 4.48,1.81 4.48,4.91 v 9.78 h -2.02 z"
id="path52" />
</g>
</g>
<rect
class="cls-20"
x="626.5166"
y="57.619633"
width="194.16"
height="62.799999"
rx="11.07"
ry="11.07"
id="rect53" />
<rect
class="cls-20"
x="626.5166"
y="152.68964"
width="194.16"
height="62.799999"
rx="11.07"
ry="11.07"
id="rect54" />
<rect
class="cls-20"
x="626.5166"
y="247.75964"
width="194.16"
height="62.799999"
rx="11.07"
ry="11.07"
id="rect55" />
<g
id="g57"
transform="translate(-147.3734,-139.91037)">
<path
class="cls-17"
d="m 854.07,237.47 c -1.26,-0.73 -2.26,-1.73 -2.98,-3 -0.72,-1.27 -1.08,-2.69 -1.08,-4.25 0,-1.56 0.36,-2.98 1.08,-4.25 0.72,-1.27 1.71,-2.27 2.98,-3 1.27,-0.73 2.67,-1.09 4.21,-1.09 1.19,0 2.28,0.22 3.27,0.67 0.99,0.45 1.85,1.09 2.58,1.94 l -1.36,1.32 c -0.61,-0.73 -1.27,-1.26 -2,-1.61 -0.72,-0.34 -1.55,-0.51 -2.49,-0.51 -1.16,0 -2.22,0.27 -3.19,0.81 -0.97,0.54 -1.74,1.31 -2.31,2.3 -0.57,0.99 -0.86,2.13 -0.86,3.42 0,1.29 0.29,2.43 0.86,3.42 0.57,0.99 1.34,1.75 2.31,2.3 0.97,0.54 2.03,0.81 3.19,0.81 1.93,0 3.58,-0.79 4.93,-2.39 l 1.38,1.34 c -0.73,0.88 -1.64,1.57 -2.73,2.08 -1.09,0.51 -2.29,0.77 -3.58,0.77 -1.55,0 -2.95,-0.36 -4.21,-1.09 z"
id="path55" />
<path
class="cls-17"
d="m 867.32,222.24 h 5.37 c 0.89,0 1.72,0.2 2.48,0.6 0.76,0.4 1.36,0.96 1.82,1.68 0.45,0.72 0.68,1.54 0.68,2.44 0,0.9 -0.23,1.72 -0.68,2.44 -0.45,0.72 -1.06,1.28 -1.82,1.68 -0.76,0.4 -1.58,0.6 -2.48,0.6 h -3.48 v 6.51 h -1.9 v -15.97 z m 5.42,7.65 c 0.59,0 1.12,-0.14 1.57,-0.42 0.45,-0.28 0.81,-0.65 1.06,-1.09 0.25,-0.45 0.38,-0.91 0.38,-1.41 0,-0.5 -0.13,-0.96 -0.38,-1.41 -0.25,-0.45 -0.61,-0.81 -1.06,-1.09 -0.45,-0.28 -0.98,-0.42 -1.57,-0.42 h -3.52 v 5.84 z"
id="path56" />
<path
class="cls-17"
d="m 882.59,237.78 c -0.89,-0.53 -1.58,-1.26 -2.07,-2.21 -0.49,-0.94 -0.74,-2.03 -0.74,-3.27 v -10.06 h 1.9 v 10.15 c 0,1.29 0.34,2.35 1.03,3.16 0.68,0.81 1.67,1.22 2.94,1.22 1.27,0 2.26,-0.41 2.95,-1.22 0.69,-0.81 1.04,-1.86 1.04,-3.16 v -10.15 h 1.9 v 10.06 c 0,1.23 -0.23,2.33 -0.7,3.28 -0.47,0.95 -1.15,1.69 -2.04,2.21 -0.89,0.52 -1.94,0.78 -3.14,0.78 -1.2,0 -2.16,-0.26 -3.06,-0.79 z"
id="path57" />
</g>
<g
id="g60"
transform="translate(-147.3734,-139.91037)">
<path
class="cls-17"
d="m 853.33,332.51 c -1.26,-0.73 -2.27,-1.73 -3.01,-3.01 -0.74,-1.28 -1.12,-2.69 -1.12,-4.24 0,-1.55 0.37,-2.96 1.12,-4.24 0.74,-1.28 1.75,-2.28 3.01,-3.01 1.26,-0.73 2.65,-1.09 4.15,-1.09 1.17,0 2.28,0.21 3.31,0.62 1.03,0.42 1.88,1 2.55,1.76 l -1.34,1.36 c -0.52,-0.62 -1.17,-1.1 -1.96,-1.44 -0.79,-0.33 -1.64,-0.5 -2.54,-0.5 -1.12,0 -2.16,0.27 -3.14,0.81 -0.98,0.54 -1.77,1.31 -2.35,2.3 -0.59,0.99 -0.88,2.13 -0.88,3.42 0,1.29 0.29,2.43 0.88,3.42 0.59,0.99 1.37,1.75 2.35,2.3 0.98,0.54 2.03,0.81 3.14,0.81 1.11,0 1.97,-0.17 2.69,-0.5 0.72,-0.33 1.34,-0.78 1.86,-1.33 0.39,-0.42 0.7,-0.92 0.94,-1.5 0.24,-0.58 0.39,-1.24 0.45,-1.95 h -5.91 v -1.76 h 7.67 c 0.07,0.42 0.11,0.8 0.11,1.16 0,0.98 -0.16,1.94 -0.47,2.87 -0.31,0.93 -0.81,1.74 -1.49,2.44 -1.47,1.59 -3.43,2.39 -5.87,2.39 -1.5,0 -2.88,-0.36 -4.15,-1.09 z"
id="path58" />
<path
class="cls-17"
d="m 868.12,317.28 h 5.37 c 0.89,0 1.72,0.2 2.48,0.6 0.76,0.4 1.36,0.96 1.82,1.68 0.45,0.72 0.68,1.54 0.68,2.44 0,0.9 -0.23,1.72 -0.68,2.44 -0.45,0.72 -1.06,1.28 -1.82,1.68 -0.76,0.4 -1.58,0.6 -2.48,0.6 h -3.48 v 6.51 h -1.9 v -15.97 z m 5.42,7.65 c 0.59,0 1.12,-0.14 1.57,-0.42 0.45,-0.28 0.81,-0.65 1.06,-1.09 0.25,-0.45 0.38,-0.91 0.38,-1.4 0,-0.49 -0.13,-0.96 -0.38,-1.41 -0.25,-0.45 -0.61,-0.81 -1.06,-1.09 -0.45,-0.28 -0.98,-0.42 -1.57,-0.42 h -3.52 v 5.84 h 3.52 z"
id="path59" />
<path
class="cls-17"
d="m 883.39,332.82 c -0.89,-0.53 -1.58,-1.26 -2.07,-2.21 -0.49,-0.94 -0.74,-2.03 -0.74,-3.27 v -10.06 h 1.9 v 10.15 c 0,1.29 0.34,2.35 1.03,3.16 0.68,0.81 1.67,1.22 2.94,1.22 1.27,0 2.26,-0.41 2.95,-1.22 0.69,-0.81 1.04,-1.86 1.04,-3.16 v -10.15 h 1.9 v 10.06 c 0,1.23 -0.23,2.33 -0.7,3.28 -0.47,0.95 -1.15,1.69 -2.04,2.21 -0.89,0.52 -1.94,0.78 -3.14,0.78 -1.2,0 -2.16,-0.26 -3.06,-0.79 z"
id="path60" />
</g>
<g
id="g63"
transform="translate(-147.3734,-139.91037)">
<path
class="cls-17"
d="m 856.14,422.03 -3.93,-5.6 h 2.14 l 2.92,4.21 2.83,-4.21 h 2.23 l -3.99,5.6 4.06,5.78 h -2.21 l -2.94,-4.37 -2.92,4.37 h -2.21 l 4.01,-5.78 z"
id="path61" />
<path
class="cls-17"
d="m 864.48,411.84 h 5.37 c 0.89,0 1.72,0.2 2.48,0.6 0.76,0.4 1.36,0.96 1.82,1.68 0.45,0.72 0.68,1.54 0.68,2.44 0,0.9 -0.23,1.72 -0.68,2.44 -0.45,0.72 -1.06,1.28 -1.82,1.68 -0.76,0.4 -1.58,0.6 -2.48,0.6 h -3.48 v 6.51 h -1.9 v -15.97 z m 5.42,7.65 c 0.59,0 1.12,-0.14 1.57,-0.42 0.45,-0.28 0.81,-0.65 1.06,-1.09 0.25,-0.45 0.38,-0.92 0.38,-1.41 0,-0.49 -0.13,-0.96 -0.38,-1.41 -0.25,-0.45 -0.61,-0.81 -1.06,-1.09 -0.45,-0.28 -0.98,-0.42 -1.57,-0.42 h -3.52 v 5.84 z"
id="path62" />
<path
class="cls-17"
d="m 879.76,427.37 c -0.89,-0.53 -1.58,-1.26 -2.07,-2.21 -0.49,-0.94 -0.74,-2.03 -0.74,-3.27 v -10.06 h 1.9 v 10.15 c 0,1.29 0.34,2.35 1.03,3.16 0.68,0.81 1.67,1.22 2.94,1.22 1.27,0 2.26,-0.41 2.95,-1.22 0.69,-0.81 1.04,-1.86 1.04,-3.16 v -10.15 h 1.9 v 10.06 c 0,1.23 -0.23,2.33 -0.7,3.28 -0.47,0.95 -1.15,1.69 -2.04,2.21 -0.89,0.52 -1.94,0.78 -3.14,0.78 -1.2,0 -2.16,-0.26 -3.06,-0.79 z"
id="path63" />
</g>
<rect
class="cls-19 openxla-box"
x="312.20657"
y="17.819626"
width="232.83"
height="332.54001"
rx="11.07"
ry="11.07"
id="rect63"
style="fill-opacity:0" />
<rect
class="cls-20 openxla-component"
x="348.00659"
y="128.83963"
width="161.24001"
height="52.150002"
rx="11.07"
ry="11.07"
id="rect64" />
<g
id="g73"
transform="translate(-147.3734,-139.91037)">
<polygon
class="cls-17"
points="532.28,233.46 506.36,233.46 494.61,213.11 506.36,192.77 532.28,192.77 544.02,213.11 "
id="polygon64" />
<polygon
class="cls-14"
points="527.77,219.7 523.97,213.11 530.87,201.16 532.42,203.84 527.07,213.11 530.87,219.7 "
id="polygon65" />
<polygon
class="cls-15"
points="510.86,206.53 514.67,213.11 507.77,225.07 506.22,222.39 511.57,213.11 507.77,206.53 "
id="polygon66" />
<polygon
class="cls-16"
points="506.22,203.84 512.42,203.84 517.77,213.11 510.87,225.07 513.97,225.07 516.22,221.17 518.47,225.07 521.57,225.07 517.77,218.48 519.32,215.8 524.67,225.07 530.87,225.07 532.42,222.39 526.22,222.39 520.87,213.11 527.77,201.16 524.67,201.16 522.42,205.06 520.17,201.16 517.07,201.16 520.87,207.74 519.32,210.43 513.97,201.16 507.77,201.16 "
id="polygon67" />
<g
id="g70">
<path
class="cls-17"
d="m 556.38,213.11 c 0,-5.25 3.32,-8.6 8.17,-8.6 4.85,0 8.14,3.35 8.14,8.6 0,5.25 -3.3,8.6 -8.14,8.6 -4.84,0 -8.17,-3.4 -8.17,-8.6 z m 13.08,0 c 0,-3.56 -1.9,-5.65 -4.92,-5.65 -3.02,0 -4.92,2.09 -4.92,5.65 0,3.56 1.9,5.67 4.92,5.67 3.02,0 4.92,-2.18 4.92,-5.67 z"
id="path67" />
<path
class="cls-17"
d="m 587.03,215.13 c 0,4.27 -2.16,6.58 -5.27,6.58 -1.42,0 -2.54,-0.5 -3.3,-1.33 v 5.72 h -3.04 v -13.8 l -1.02,-2.07 2.94,-1.5 0.83,1.73 c 0.85,-1.02 2.16,-1.85 3.92,-1.85 3.06,0 4.94,2.49 4.94,6.51 z m -3.13,-0.02 c 0,-2.37 -1.07,-3.78 -3.13,-3.78 -0.83,0 -1.64,0.21 -2.3,0.59 v 6.39 c 0.64,0.43 1.42,0.66 2.23,0.66 1.99,0 3.21,-1.38 3.21,-3.87 z"
id="path68" />
<path
class="cls-17"
d="m 599.97,215.63 h -8.12 c 0.26,1.92 1.76,3.11 4.23,3.11 1.35,0 2.83,-0.29 3.8,-0.81 l -0.93,3.04 c -0.85,0.47 -2.14,0.74 -3.68,0.74 -4.13,0 -6.58,-2.59 -6.58,-6.58 0,-3.99 2.4,-6.51 6.29,-6.51 3.3,0 5.3,1.88 5.3,4.92 0,0.66 -0.09,1.5 -0.31,2.09 z m -8.05,-1.9 h 5.44 c -0.1,-1.83 -0.93,-2.61 -2.37,-2.61 -1.61,0 -2.71,0.97 -3.06,2.61 z"
id="path69" />
<path
class="cls-17"
d="m 613.46,212.66 v 8.74 h -3.04 v -7.53 c 0,-1.64 -0.66,-2.49 -2.23,-2.49 -0.9,0 -1.73,0.31 -2.42,0.62 v 9.4 h -3.04 v -9.12 l -1.02,-2.18 2.94,-1.4 0.95,1.95 c 1.38,-1.23 2.8,-1.99 4.25,-1.99 2.33,0 3.61,1.38 3.61,4.01 z"
id="path70" />
</g>
<path
class="cls-15"
d="m 629.35,204.83 -5.06,8.52 5.08,8.05 h -3.61 l -2.92,-4.7 -0.59,-1.38 h -0.14 l -0.55,1.35 -2.75,4.73 h -3.51 l 5.2,-8.38 -5.13,-8.19 h 3.56 l 3.06,4.92 0.5,1.26 h 0.14 l 0.55,-1.31 2.73,-4.87 z"
id="path71" />
<path
class="cls-15"
d="m 634.64,218.72 h 5.84 v 2.68 h -8.93 v -16.57 h 3.09 z"
id="path72" />
<path
class="cls-15"
d="m 652.67,221.4 -1.33,-4.23 h -5.72 l -1.35,4.23 h -3.13 l 5.41,-16.57 h 3.92 l 5.39,16.57 h -3.18 z m -6.24,-6.77 h 4.11 l -1.71,-5.41 -0.26,-1.28 h -0.17 l -0.26,1.28 z"
id="path73" />
</g>
<g
id="g82"
transform="translate(-147.3734,-139.91037)">
<path
class="cls-17 openxla-component-text"
d="m 529.75,303.12 c -0.7,-0.31 -1.32,-0.77 -1.86,-1.37 -0.54,-0.6 -0.93,-1.33 -1.16,-2.19 l 1.68,-0.69 c 0.24,0.89 0.67,1.61 1.31,2.18 0.64,0.57 1.38,0.85 2.24,0.85 0.86,0 1.58,-0.23 2.18,-0.7 0.59,-0.46 0.89,-1.1 0.89,-1.9 0,-0.69 -0.25,-1.26 -0.76,-1.71 -0.51,-0.45 -1.33,-0.87 -2.48,-1.26 L 530.86,296 c -1.05,-0.37 -1.91,-0.86 -2.57,-1.47 -0.66,-0.61 -1,-1.45 -1,-2.53 0,-0.71 0.19,-1.36 0.58,-1.96 0.39,-0.6 0.93,-1.08 1.62,-1.44 0.69,-0.36 1.47,-0.54 2.33,-0.54 0.86,0 1.61,0.16 2.24,0.48 0.64,0.32 1.15,0.71 1.53,1.16 0.38,0.46 0.65,0.91 0.8,1.35 l -1.64,0.71 c -0.15,-0.51 -0.47,-0.98 -0.97,-1.39 -0.49,-0.42 -1.14,-0.62 -1.94,-0.62 -0.76,0 -1.41,0.21 -1.95,0.64 -0.54,0.43 -0.81,0.97 -0.81,1.62 0,0.58 0.23,1.06 0.69,1.42 0.46,0.37 1.15,0.71 2.08,1.03 l 0.98,0.33 c 1.26,0.46 2.24,1.03 2.95,1.71 0.71,0.68 1.06,1.62 1.06,2.79 0,0.96 -0.25,1.76 -0.74,2.41 -0.49,0.65 -1.11,1.13 -1.86,1.43 -0.75,0.3 -1.52,0.46 -2.3,0.46 -0.75,0 -1.47,-0.16 -2.17,-0.47 z"
id="path74" />
<path
class="cls-17 openxla-component-text"
d="m 541.44,303.38 c -0.37,-0.14 -0.67,-0.33 -0.92,-0.57 -0.55,-0.54 -0.83,-1.27 -0.83,-2.22 v -6.32 h -1.85 v -1.6 h 1.85 v -2.99 h 1.76 v 2.99 h 2.6 v 1.6 h -2.6 v 5.9 c 0,0.59 0.11,1.03 0.33,1.32 0.26,0.31 0.64,0.47 1.14,0.47 0.43,0 0.82,-0.12 1.16,-0.35 v 1.72 c -0.21,0.1 -0.42,0.17 -0.63,0.21 -0.21,0.04 -0.49,0.06 -0.82,0.06 -0.43,0 -0.83,-0.07 -1.19,-0.21 z"
id="path75" />
<path
class="cls-17 openxla-component-text"
d="m 547.27,303.13 c -0.61,-0.3 -1.08,-0.72 -1.41,-1.26 -0.33,-0.53 -0.5,-1.14 -0.5,-1.82 0,-1.12 0.42,-2 1.27,-2.63 0.84,-0.63 1.91,-0.95 3.2,-0.95 0.64,0 1.23,0.07 1.77,0.21 0.54,0.14 0.97,0.3 1.26,0.48 v -0.64 c 0,-0.79 -0.28,-1.42 -0.83,-1.9 -0.55,-0.48 -1.25,-0.72 -2.1,-0.72 -0.58,0 -1.12,0.13 -1.63,0.38 -0.51,0.26 -0.9,0.61 -1.19,1.07 l -1.33,-1 c 0.42,-0.64 0.99,-1.13 1.72,-1.5 0.73,-0.36 1.54,-0.54 2.43,-0.54 1.44,0 2.57,0.38 3.38,1.13 0.81,0.75 1.23,1.78 1.23,3.08 v 6.71 h -1.68 v -1.52 h -0.08 c -0.3,0.51 -0.76,0.95 -1.37,1.31 -0.61,0.36 -1.29,0.54 -2.06,0.54 -0.77,0 -1.47,-0.15 -2.08,-0.46 z m 3.9,-1.56 c 0.51,-0.3 0.92,-0.71 1.22,-1.22 0.3,-0.51 0.46,-1.07 0.46,-1.68 -0.33,-0.22 -0.74,-0.4 -1.23,-0.54 -0.49,-0.14 -1,-0.21 -1.54,-0.21 -0.97,0 -1.7,0.2 -2.19,0.6 -0.49,0.4 -0.74,0.92 -0.74,1.56 0,0.58 0.22,1.05 0.66,1.41 0.44,0.36 1,0.54 1.68,0.54 0.6,0 1.15,-0.15 1.66,-0.46 z"
id="path76" />
<path
class="cls-17 openxla-component-text"
d="m 560.19,303.04 c -0.64,-0.37 -1.12,-0.81 -1.44,-1.34 h -0.08 v 1.56 h -1.68 v -14.87 h 1.76 v 4.38 l -0.08,1.47 h 0.08 c 0.32,-0.54 0.8,-0.99 1.44,-1.36 0.64,-0.37 1.37,-0.55 2.17,-0.55 0.96,0 1.82,0.24 2.58,0.73 0.77,0.49 1.37,1.16 1.82,2.01 0.45,0.85 0.66,1.82 0.66,2.89 0,1.07 -0.22,2.04 -0.66,2.9 -0.44,0.85 -1.05,1.52 -1.82,2 -0.77,0.48 -1.63,0.73 -2.58,0.73 -0.8,0 -1.53,-0.18 -2.17,-0.55 z m 3.71,-1.55 c 0.54,-0.33 0.97,-0.8 1.29,-1.41 0.32,-0.61 0.48,-1.31 0.48,-2.12 0,-0.81 -0.16,-1.51 -0.48,-2.12 -0.32,-0.61 -0.75,-1.08 -1.29,-1.41 -0.54,-0.33 -1.11,-0.5 -1.72,-0.5 -0.61,0 -1.2,0.17 -1.73,0.5 -0.53,0.33 -0.96,0.8 -1.29,1.41 -0.33,0.61 -0.49,1.32 -0.49,2.12 0,0.8 0.16,1.51 0.49,2.12 0.33,0.61 0.75,1.08 1.29,1.41 0.53,0.33 1.11,0.5 1.73,0.5 0.62,0 1.18,-0.17 1.72,-0.5 z"
id="path77" />
<path
class="cls-17 openxla-component-text"
d="m 569.38,288.39 h 1.76 v 14.87 h -1.76 z"
id="path78" />
<path
class="cls-17 openxla-component-text"
d="m 575.65,302.86 c -0.8,-0.48 -1.43,-1.15 -1.88,-2 -0.45,-0.85 -0.67,-1.81 -0.67,-2.88 0,-1.01 0.21,-1.95 0.63,-2.81 0.42,-0.86 1.02,-1.55 1.81,-2.07 0.78,-0.51 1.68,-0.77 2.69,-0.77 1.01,0 1.94,0.23 2.7,0.7 0.76,0.46 1.35,1.1 1.75,1.92 0.4,0.82 0.61,1.75 0.61,2.8 0,0.17 -0.01,0.35 -0.04,0.54 h -8.39 c 0.04,0.8 0.24,1.48 0.58,2.03 0.35,0.55 0.79,0.97 1.32,1.25 0.53,0.28 1.09,0.42 1.67,0.42 1.38,0 2.43,-0.64 3.14,-1.91 l 1.49,0.73 c -0.44,0.84 -1.06,1.52 -1.85,2.02 -0.79,0.51 -1.74,0.76 -2.84,0.76 -1.01,0 -1.92,-0.24 -2.72,-0.73 z m 5.77,-6.02 c -0.03,-0.44 -0.15,-0.89 -0.37,-1.33 -0.22,-0.44 -0.57,-0.82 -1.04,-1.12 -0.47,-0.3 -1.07,-0.46 -1.79,-0.46 -0.83,0 -1.54,0.27 -2.12,0.8 -0.58,0.53 -0.96,1.24 -1.14,2.11 z"
id="path79" />
<path
class="cls-17 openxla-component-text"
d="m 585.7,288.39 h 1.76 v 6.42 h 7.64 v -6.42 h 1.76 v 14.87 h -1.76 v -6.77 h -7.64 v 6.77 h -1.76 z"
id="path80" />
<path
class="cls-17 openxla-component-text"
d="m 600.19,288.39 h 1.76 v 13.18 h 6.5 v 1.68 h -8.26 v -14.87 z"
id="path81" />
<path
class="cls-17 openxla-component-text"
d="m 613.46,302.55 c -1.16,-0.69 -2.08,-1.63 -2.74,-2.82 -0.66,-1.19 -1,-2.49 -1,-3.9 0,-1.41 0.33,-2.71 1,-3.9 0.66,-1.19 1.58,-2.13 2.74,-2.82 1.16,-0.69 2.46,-1.04 3.88,-1.04 1.42,0 2.72,0.35 3.88,1.04 1.16,0.69 2.08,1.63 2.74,2.82 0.66,1.19 1,2.49 1,3.9 0,1.41 -0.33,2.71 -1,3.9 -0.67,1.19 -1.58,2.13 -2.74,2.82 -1.16,0.69 -2.46,1.04 -3.88,1.04 -1.42,0 -2.72,-0.35 -3.88,-1.04 z m 6.82,-1.42 c 0.89,-0.52 1.6,-1.24 2.12,-2.16 0.52,-0.92 0.78,-1.97 0.78,-3.15 0,-1.18 -0.26,-2.22 -0.78,-3.15 -0.52,-0.93 -1.23,-1.64 -2.12,-2.16 -0.89,-0.52 -1.87,-0.78 -2.94,-0.78 -1.07,0 -2.05,0.26 -2.94,0.78 -0.89,0.52 -1.6,1.24 -2.12,2.16 -0.52,0.92 -0.78,1.97 -0.78,3.15 0,1.18 0.26,2.22 0.78,3.15 0.52,0.93 1.23,1.64 2.12,2.16 0.89,0.52 1.87,0.78 2.94,0.78 1.07,0 2.04,-0.26 2.94,-0.78 z"
id="path82" />
</g>
<rect
class="cls-20 openxla-component"
x="348.00659"
y="195.56964"
width="161.24001"
height="52.150002"
rx="11.07"
ry="11.07"
id="rect82" />
<g
id="g85"
transform="translate(-147.3734,-139.91037)">
<path
class="cls-17 openxla-component-text"
d="m 562.61,362.19 -4.73,-7.18 H 560 l 3.74,5.79 h 0.08 l 3.74,-5.79 h 2.1 l -4.73,7.18 5.02,7.68 h -2.08 l -4.05,-6.21 h -0.08 l -4.07,6.21 h -2.08 z"
id="path83" />
<path
class="cls-17 openxla-component-text"
d="m 571.97,355.01 h 1.76 v 13.18 h 6.5 v 1.68 h -8.26 V 355 Z"
id="path84" />
<path
class="cls-17 openxla-component-text"
d="m 586.84,355.01 h 1.99 l 5.65,14.87 h -1.93 l -1.5,-4.09 h -6.44 l -1.5,4.09 h -1.93 l 5.65,-14.87 z m 3.61,9.12 -1.99,-5.36 -0.58,-1.6 h -0.08 l -0.58,1.6 -1.99,5.36 h 5.23 z"
id="path85" />
</g>
<rect
class="cls-20 openxla-component"
x="348.00659"
y="262.29962"
width="161.24001"
height="52.150002"
rx="11.07"
ry="11.07"
id="rect85" />
<g
id="g89"
transform="translate(-147.3734,-139.91037)">
<path
class="cls-17 openxla-component-text"
d="m 555.86,421.85 h 5 c 0.83,0 1.6,0.19 2.3,0.56 0.71,0.37 1.27,0.9 1.69,1.57 0.42,0.67 0.63,1.43 0.63,2.27 0,0.84 -0.21,1.6 -0.63,2.27 -0.42,0.67 -0.99,1.19 -1.69,1.57 -0.71,0.37 -1.47,0.56 -2.3,0.56 h -3.24 v 6.06 h -1.76 v -14.87 z m 5.04,7.12 c 0.55,0 1.04,-0.13 1.46,-0.39 0.42,-0.26 0.75,-0.6 0.99,-1.02 0.24,-0.42 0.35,-0.85 0.35,-1.31 0,-0.46 -0.12,-0.89 -0.35,-1.31 -0.23,-0.42 -0.56,-0.75 -0.99,-1.02 -0.42,-0.26 -0.91,-0.39 -1.46,-0.39 h -3.28 v 5.44 z"
id="path86" />
<path
class="cls-17 openxla-component-text"
d="m 565.72,436.14 c -0.82,-0.6 -1.39,-1.45 -1.72,-2.54 l 1.66,-0.69 c 0.5,1.63 1.47,2.45 2.93,2.45 0.82,0 1.47,-0.28 1.96,-0.85 0.49,-0.57 0.74,-1.31 0.74,-2.24 v -10.42 h 1.76 v 10.28 c 0,1.04 -0.19,1.93 -0.57,2.67 -0.38,0.74 -0.91,1.3 -1.58,1.68 -0.67,0.38 -1.44,0.57 -2.31,0.57 -1.09,0 -2.05,-0.3 -2.87,-0.9 z"
id="path87" />
<path
class="cls-17 openxla-component-text"
d="m 576.13,421.85 h 5.02 c 0.82,0 1.58,0.19 2.28,0.56 0.71,0.37 1.27,0.9 1.69,1.57 0.42,0.67 0.63,1.43 0.63,2.27 0,0.69 -0.16,1.34 -0.47,1.93 -0.31,0.59 -0.74,1.09 -1.29,1.5 -0.55,0.4 -1.15,0.67 -1.82,0.81 l -0.04,0.06 4.19,6.08 v 0.08 h -2.08 l -4.05,-6.06 h -2.33 v 6.06 h -1.76 v -14.87 z m 4.92,7.16 c 0.51,0 0.99,-0.12 1.44,-0.35 0.45,-0.24 0.81,-0.56 1.08,-0.99 0.27,-0.42 0.4,-0.9 0.4,-1.42 0,-0.46 -0.12,-0.89 -0.35,-1.31 -0.23,-0.42 -0.56,-0.75 -0.99,-1.02 -0.42,-0.26 -0.91,-0.39 -1.46,-0.39 h -3.28 v 5.48 z"
id="path88" />
<path
class="cls-17 openxla-component-text"
d="m 591.34,423.53 h -4.17 v -1.68 h 10.09 v 1.68 h -4.15 v 13.18 h -1.77 z"
id="path89" />
</g>
<line
class="cls-21"
x1="230.73657"
y1="184.08963"
x2="312.20657"
y2="184.08963"
id="line89" />
<path
class="cls-21"
d="m 230.7366,89.019631 h 29.66 c 6.12,0 11.07,4.96 11.07,11.069999 v 167.88 c 0,6.12 -4.96,11.07 -11.07,11.07 h -29.66"
id="path90" />
<line
class="cls-21"
x1="626.5166"
y1="183.97964"
x2="545.04657"
y2="183.97964"
id="line90" />
<path
class="cls-21"
d="m 626.5166,279.04963 h -29.66 c -6.12,0 -11.07,-4.96 -11.07,-11.07 v -167.88 c 0,-6.119999 4.96,-11.069999 11.07,-11.069999 h 29.66"
id="path91" />
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 34 KiB

third_party/xla/docs/images/openxla_dark.svg
Vendored
+714
View File
@@ -0,0 +1,714 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<svg
id="Layer_1"
data-name="Layer 1"
viewBox="0 0 780 335"
version="1.1"
sodipodi:docname="openxla_dark.svg"
width="780"
height="335"
inkscape:version="1.4 (e7c3feb1, 2024-10-09)"
xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns="http://www.w3.org/2000/svg"
xmlns:svg="http://www.w3.org/2000/svg">
<sodipodi:namedview
id="namedview91"
pagecolor="#868074"
bordercolor="#000000"
borderopacity="0.25"
inkscape:showpageshadow="2"
inkscape:pageopacity="0.0"
inkscape:pagecheckerboard="0"
inkscape:deskcolor="#d1d1d1"
inkscape:zoom="0.57723979"
inkscape:cx="343.87789"
inkscape:cy="89.217688"
inkscape:window-width="1472"
inkscape:window-height="636"
inkscape:window-x="0"
inkscape:window-y="38"
inkscape:window-maximized="0"
inkscape:current-layer="Layer_1" />
<defs
id="defs2">
<style
id="style1">
.cls-1 {
fill: #ea80fc;
}
.cls-1, .cls-2, .cls-3, .cls-4, .cls-5, .cls-6, .cls-7, .cls-8, .cls-9 {
stroke: #dce0df;
stroke-linejoin: round;
stroke-width: .13px;
}
.cls-2 {
fill: #3367d6;
}
.cls-3 {
fill: #5e97f6;
}
.cls-4 {
fill: #26a69a;
}
.cls-5 {
fill: #2a56c6;
}
.cls-6 {
fill: #9c27b0;
}
.cls-7 {
fill: #6a1b9a;
}
.cls-8 {
fill: #00695c;
}
.cls-9 {
fill: #00796b;
}
.cls-10 {
clip-path: url(#clippath);
}
.cls-11, .cls-12, .cls-13, .cls-14, .cls-15, .cls-16, .cls-17, .cls-18 {
stroke-width: 0px;
}
.cls-11, .cls-19, .cls-20, .cls-21 {
fill: none;
}
.cls-12 {
fill: url(#linear-gradient);
}
.cls-19 {
stroke-width: .65px;
}
.cls-19, .cls-20, .cls-21 {
stroke: #fff;
stroke-miterlimit: 10;
}
.cls-20 {
stroke-width: .65px;
}
.cls-21 {
stroke-width: .54px;
}
.cls-22 {
clip-path: url(#clippath-1);
}
.cls-13 {
fill: url(#linear-gradient-2);
}
.cls-14 {
fill: #458ae5;
}
.cls-15 {
fill: #6fb6e0;
}
.cls-16 {
fill: #1a344c;
}
.cls-17 {
fill: #fff;
}
.cls-18 {
fill: #ee4c2c;
}
</style>
<clipPath
id="clippath">
<polygon
class="cls-11"
points="231.81,231.78 234.89,233.54 234.84,228.97 231.81,227.21 231.81,224.54 237.92,228.11 237.92,222.82 227.74,217.03 227.74,240.83 231.81,238.47 "
id="polygon1" />
</clipPath>
<linearGradient
id="linear-gradient"
x1="-348.29001"
y1="-1465.28"
x2="-323.32001"
y2="-1465.28"
gradientTransform="matrix(1,0,0,-1,564,-1236.37)"
gradientUnits="userSpaceOnUse">
<stop
offset="0"
stop-color="#ff6f00"
id="stop1" />
<stop
offset="1"
stop-color="#ffa800"
id="stop2" />
</linearGradient>
<clipPath
id="clippath-1">
<polygon
class="cls-11"
points="222.76,224.54 216.66,228.11 216.66,222.82 226.83,217.03 226.83,240.83 222.76,238.47 "
id="polygon2" />
</clipPath>
<linearGradient
id="linear-gradient-2"
x1="-348.42999"
x2="-323.45999"
xlink:href="#linear-gradient" />
<linearGradient
inkscape:collect="always"
xlink:href="#linear-gradient"
id="linearGradient91"
gradientUnits="userSpaceOnUse"
gradientTransform="matrix(1,0,0,-1,564,-1236.37)"
x1="-348.29001"
y1="-1465.28"
x2="-323.32001"
y2="-1465.28" />
</defs>
<rect
class="cls-20"
x="20.373598"
y="48.263557"
width="183.05487"
height="59.208103"
rx="10.436842"
ry="10.436842"
id="rect2" />
<g
id="Layer_2"
data-name="Layer 2"
transform="matrix(0.9428042,0,0,0.9428042,-153.05524,-137.96856)">
<g
id="Layer_1-2"
data-name="Layer 1-2">
<polygon
class="cls-3"
points="257.34,324 254.17,329.48 260.5,329.48 263.67,324 "
id="polygon3" />
<polygon
class="cls-3"
points="251.01,334.96 254.17,329.48 260.5,329.48 257.34,334.96 "
id="polygon4" />
<polygon
class="cls-3"
points="266.83,329.48 260.5,329.48 257.34,334.96 263.67,334.96 "
id="polygon5" />
<polygon
class="cls-3"
points="273.16,329.48 266.83,329.48 263.67,334.96 269.99,334.96 "
id="polygon6" />
<polygon
class="cls-3"
points="269.99,324 266.83,329.48 273.16,329.48 276.32,324 "
id="polygon7" />
<polygon
class="cls-3"
points="273.16,318.52 269.99,324 276.32,324 279.49,318.52 "
id="polygon8" />
<polygon
class="cls-3"
points="276.32,313.04 273.16,318.52 279.49,318.52 282.65,313.04 "
id="polygon9" />
<polygon
class="cls-3"
points="279.49,307.56 276.32,313.04 282.65,313.04 285.82,307.56 "
id="polygon10" />
<polygon
class="cls-5"
points="251.01,334.96 254.17,340.44 260.5,340.44 257.34,334.96 "
id="polygon11" />
<polygon
class="cls-5"
points="266.83,340.44 260.5,340.44 257.34,334.96 263.67,334.96 "
id="polygon12" />
<polygon
class="cls-5"
points="273.16,340.44 266.83,340.44 263.67,334.96 269.99,334.96 "
id="polygon13" />
<polygon
class="cls-9"
points="276.32,334.96 273.16,329.48 269.99,334.96 273.16,340.44 "
id="polygon14" />
<polygon
class="cls-9"
points="282.65,324 279.49,318.52 276.32,324 "
id="polygon15" />
<polygon
class="cls-9"
points="282.65,313.04 279.49,318.52 282.65,324 285.82,318.52 "
id="polygon16" />
<polygon
class="cls-2"
points="266.83,329.48 263.67,324 260.5,329.48 "
id="polygon17" />
<polygon
class="cls-4"
points="282.65,324 276.32,324 273.16,329.48 279.49,329.48 "
id="polygon18" />
<polygon
class="cls-4"
points="288.98,324 282.65,324 279.49,329.48 285.82,329.48 "
id="polygon19" />
<polygon
class="cls-6"
points="295.31,313.04 292.15,307.56 288.98,313.04 292.15,318.52 "
id="polygon20" />
<polygon
class="cls-6"
points="298.48,318.52 295.31,313.04 292.15,318.52 295.31,324 "
id="polygon21" />
<polygon
class="cls-6"
points="301.64,324 298.48,318.52 295.31,324 298.48,329.48 "
id="polygon22" />
<polygon
class="cls-6"
points="304.81,329.48 301.64,324 298.48,329.48 301.64,334.96 "
id="polygon23" />
<polygon
class="cls-6"
points="307.97,334.96 304.81,329.48 301.64,334.96 304.81,340.44 "
id="polygon24" />
<polygon
class="cls-6"
points="304.81,307.56 301.64,313.04 304.81,318.52 307.97,313.04 "
id="polygon25" />
<polygon
class="cls-6"
points="298.48,318.52 301.64,313.04 304.81,318.52 301.64,324 "
id="polygon26" />
<polygon
class="cls-6"
points="295.31,324 292.15,329.48 295.31,334.96 298.48,329.48 "
id="polygon27" />
<polygon
class="cls-6"
points="292.15,340.44 288.98,334.96 292.15,329.48 295.31,334.96 "
id="polygon28" />
<polygon
class="cls-7"
points="285.82,340.44 282.65,334.96 288.98,334.96 292.15,340.44 "
id="polygon29" />
<polygon
class="cls-8"
points="279.49,329.48 273.16,329.48 276.32,334.96 282.65,334.96 "
id="polygon30" />
<polygon
class="cls-8"
points="285.82,329.48 279.49,329.48 282.65,334.96 "
id="polygon31" />
<polygon
class="cls-8"
points="285.82,318.52 288.98,324 295.31,324 292.15,318.52 "
id="polygon32" />
<polygon
class="cls-8"
points="288.98,313.04 282.65,313.04 285.82,318.52 292.15,318.52 "
id="polygon33" />
<polygon
class="cls-8"
points="304.81,340.44 301.64,334.96 295.31,334.96 298.48,340.44 "
id="polygon34" />
<polygon
class="cls-8"
points="298.48,329.48 295.31,334.96 301.64,334.96 "
id="polygon35" />
<polygon
class="cls-1"
points="292.15,307.56 285.82,307.56 282.65,313.04 288.98,313.04 "
id="polygon36" />
<polygon
class="cls-1"
points="292.15,329.48 285.82,329.48 282.65,334.96 288.98,334.96 "
id="polygon37" />
<polygon
class="cls-1"
points="295.31,324 288.98,324 285.82,329.48 292.15,329.48 "
id="polygon38" />
<polygon
class="cls-1"
points="304.81,307.56 298.48,307.56 295.31,313.04 301.64,313.04 "
id="polygon39" />
<polygon
class="cls-1"
points="298.48,318.52 295.31,313.04 301.64,313.04 "
id="polygon40" />
</g>
</g>
<g
id="g44"
transform="matrix(0.9428042,0,0,0.9428042,-153.05524,-137.96856)">
<g
class="cls-10"
clip-path="url(#clippath)"
id="g40">
<path
class="cls-12"
d="m 215.71,216.94 h 24.97 v 23.93 h -24.97 z"
id="path40"
style="fill:url(#linearGradient91)" />
</g>
<g
class="cls-22"
clip-path="url(#clippath-1)"
id="g41">
<path
class="cls-13"
d="m 215.57,216.94 h 24.97 v 23.93 h -24.97 z"
id="path41"
style="fill:url(#linear-gradient-2)" />
</g>
<g
id="g43">
<path
class="cls-17"
d="m 256.33,224.41 h -4.52 v 12.53 h -2.53 v -12.53 h -4.52 v -2.04 h 11.58 v 2.04 z"
id="path42" />
<path
class="cls-17"
d="m 259.36,237.12 c -1.54,0 -2.8,-0.5 -3.75,-1.45 -0.95,-0.95 -1.45,-2.26 -1.45,-3.89 v -0.32 c 0,-1 0.18,-1.99 0.63,-2.9 0.41,-0.81 1,-1.49 1.76,-1.99 0.77,-0.5 1.63,-0.72 2.53,-0.72 1.49,0 2.62,0.45 3.44,1.4 0.82,0.95 1.22,2.26 1.22,3.98 v 1 h -7.1 c 0.05,0.81 0.36,1.54 0.9,2.13 0.54,0.54 1.22,0.81 1.99,0.77 1.09,0.05 2.08,-0.5 2.71,-1.36 l 1.31,1.27 c -0.45,0.63 -1.04,1.18 -1.72,1.49 -0.81,0.45 -1.63,0.63 -2.49,0.59 z m -0.27,-9.27 c -0.63,-0.05 -1.22,0.23 -1.63,0.68 -0.45,0.54 -0.72,1.22 -0.77,1.95 h 4.66 v -0.18 c -0.05,-0.81 -0.27,-1.45 -0.63,-1.85 -0.45,-0.36 -1.04,-0.63 -1.63,-0.59 z m 8.78,-1.76 0.09,1.27 c 0.77,-0.95 1.95,-1.49 3.17,-1.45 2.26,0 3.39,1.31 3.44,3.89 v 7.15 h -2.44 v -7.01 c 0,-0.68 -0.14,-1.18 -0.45,-1.54 -0.32,-0.32 -0.77,-0.5 -1.45,-0.5 -0.95,-0.05 -1.81,0.5 -2.22,1.31 v 7.69 h -2.44 v -10.86 c 0,0 2.31,0.05 2.31,0.05 z m 14.57,7.92 c 0,-0.41 -0.18,-0.77 -0.54,-1 -0.54,-0.32 -1.18,-0.5 -1.76,-0.59 -0.72,-0.14 -1.4,-0.36 -2.08,-0.68 -1.22,-0.59 -1.81,-1.45 -1.81,-2.53 0,-0.9 0.45,-1.81 1.18,-2.35 0.77,-0.63 1.81,-0.95 2.99,-0.95 1.31,0 2.35,0.32 3.12,0.95 0.77,0.59 1.22,1.54 1.18,2.49 h -2.44 c 0,-0.45 -0.18,-0.86 -0.54,-1.18 -0.41,-0.32 -0.86,-0.5 -1.4,-0.45 -0.45,0 -0.9,0.09 -1.31,0.36 -0.32,0.23 -0.5,0.59 -0.5,1 0,0.36 0.18,0.68 0.45,0.86 0.32,0.23 0.95,0.41 1.9,0.63 0.77,0.14 1.54,0.41 2.26,0.77 0.5,0.23 0.9,0.59 1.22,1.04 0.27,0.45 0.41,0.95 0.41,1.49 0,0.95 -0.45,1.81 -1.22,2.35 -0.81,0.59 -1.85,0.9 -3.17,0.9 -0.81,0 -1.63,-0.14 -2.35,-0.5 -0.63,-0.27 -1.22,-0.72 -1.63,-1.31 -0.36,-0.54 -0.59,-1.18 -0.59,-1.81 h 2.35 c 0,0.5 0.23,1 0.63,1.31 0.45,0.32 1.04,0.5 1.58,0.45 0.63,0 1.13,-0.14 1.45,-0.36 0.45,-0.18 0.63,-0.54 0.63,-0.9 v 0 z m 3.66,-2.58 c 0,-1 0.18,-1.99 0.63,-2.85 0.41,-0.81 1,-1.49 1.76,-1.95 0.81,-0.45 1.72,-0.72 2.62,-0.68 1.45,0 2.67,0.45 3.57,1.4 0.9,0.95 1.4,2.17 1.49,3.75 v 0.59 c 0,1 -0.18,1.95 -0.63,2.85 -0.36,0.81 -1,1.49 -1.76,1.95 -0.81,0.45 -1.72,0.72 -2.67,0.68 -1.54,0 -2.76,-0.5 -3.66,-1.54 -0.9,-1 -1.36,-2.35 -1.4,-4.07 l 0.05,-0.14 z m 2.4,0.23 c 0,1.13 0.23,1.99 0.68,2.62 0.81,1.04 2.31,1.27 3.39,0.45 0.18,-0.14 0.32,-0.27 0.45,-0.45 0.45,-0.63 0.68,-1.58 0.68,-2.8 0,-1.09 -0.23,-1.95 -0.72,-2.62 -0.77,-1.04 -2.26,-1.27 -3.35,-0.5 -0.18,0.14 -0.36,0.32 -0.5,0.45 -0.36,0.63 -0.63,1.58 -0.63,2.85 z m 14.97,-3.3 c -0.32,-0.05 -0.68,-0.09 -1,-0.09 -1.13,0 -1.85,0.41 -2.26,1.27 v 7.42 h -2.44 V 226.1 h 2.31 l 0.05,1.22 c 0.59,-0.95 1.4,-1.4 2.44,-1.4 0.27,0 0.59,0.05 0.86,0.14 l 0.05,2.31 v 0 z m 10.18,2.4 h -5.88 v 6.2 h -2.53 v -14.57 h 9.27 v 2.04 h -6.74 v 4.34 h 5.88 z m 4.84,6.2 h -2.44 v -14.61 h 2.44 z m 1.76,-5.52 c 0,-1 0.18,-1.99 0.63,-2.85 0.41,-0.81 1,-1.49 1.76,-1.95 0.81,-0.45 1.72,-0.72 2.62,-0.68 1.45,0 2.67,0.45 3.57,1.4 0.9,0.95 1.4,2.17 1.49,3.75 v 0.59 c 0,1 -0.18,1.95 -0.59,2.85 -0.36,0.81 -1,1.49 -1.76,1.95 -0.81,0.45 -1.72,0.72 -2.67,0.68 -1.54,0 -2.76,-0.5 -3.66,-1.54 -0.9,-1 -1.4,-2.35 -1.4,-4.07 v -0.14 0 z m 2.44,0.23 c 0,1.13 0.23,1.99 0.68,2.62 0.45,0.63 1.18,1 1.95,0.95 0.77,0.05 1.49,-0.32 1.9,-0.95 0.45,-0.63 0.68,-1.58 0.68,-2.8 0,-1.09 -0.23,-1.95 -0.72,-2.62 -0.77,-1.04 -2.26,-1.27 -3.35,-0.5 -0.18,0.14 -0.36,0.32 -0.5,0.45 -0.41,0.63 -0.63,1.58 -0.63,2.85 z m 18.64,1.95 1.72,-7.46 h 2.35 l -2.94,10.86 h -1.99 l -2.31,-7.46 -2.31,7.46 h -1.99 l -2.99,-10.86 h 2.4 l 1.76,7.42 2.22,-7.42 h 1.85 l 2.22,7.46 v 0 z"
id="path43" />
</g>
</g>
<rect
class="cls-20"
x="20.373598"
y="137.89595"
width="183.05487"
height="59.208103"
rx="10.436842"
ry="10.436842"
id="rect44" />
<path
class="cls-20"
d="M 192.9822,286.73645 H 30.810444 c -5.769961,0 -10.436842,-4.67631 -10.436842,-10.43684 v -38.33442 c 0,-5.76996 4.676309,-10.43685 10.436842,-10.43685 H 192.9822 c 5.76996,0 10.43684,4.67631 10.43684,10.43685 v 38.33442 c 0,5.76996 -4.67631,10.43684 -10.43684,10.43684 z"
id="path44" />
<g
id="g53"
transform="matrix(0.9428042,0,0,0.9428042,-153.05524,-137.96856)">
<g
id="g45">
<path
class="cls-18"
d="m 237.41,412.75 -1.59,1.59 c 2.6,2.6 2.6,6.79 0,9.34 -2.6,2.6 -6.79,2.6 -9.34,0 -2.6,-2.6 -2.6,-6.79 0,-9.34 v 0 l 4.12,-4.12 0.58,-0.58 v 0 -3.11 l -6.21,6.21 c -3.47,3.47 -3.47,9.05 0,12.52 3.47,3.47 9.05,3.47 12.45,0 3.47,-3.49 3.47,-9.05 0,-12.52 z"
id="path45" />
<circle
class="cls-18"
cx="234.3"
cy="411.20999"
r="1.16"
id="circle45" />
</g>
<g
id="g52">
<path
class="cls-17"
d="m 253.47,420.69 h -2.67 v 6.86 h -2.02 v -19.53 h 4.91 c 5.13,0 7.58,2.53 7.58,6.07 0.07,4.36 -2.96,6.6 -7.8,6.6 z m 0.22,-10.72 h -2.82 v 8.98 l 2.75,-0.07 c 3.68,-0.07 5.71,-1.52 5.71,-4.55 0,-2.84 -2.02,-4.36 -5.63,-4.36 z"
id="path46" />
<path
class="cls-17"
d="m 270.32,427.48 -1.16,3.11 c -1.3,3.47 -2.67,4.48 -4.62,4.48 -1.08,0 -1.88,-0.29 -2.75,-0.65 l 0.58,-1.81 c 0.65,0.36 1.37,0.65 2.17,0.65 1.08,0 1.88,-0.58 2.96,-3.32 l 0.94,-2.53 -5.56,-14.11 h 2.09 l 4.48,11.8 4.41,-11.8 h 2.02 z"
id="path47" />
<path
class="cls-17"
d="m 282.48,409.98 v 17.65 h -2.02 v -17.65 h -6.86 v -1.88 h 15.7 v 1.88 h -6.81 z"
id="path48" />
<path
class="cls-17"
d="m 295.01,427.99 c -3.97,0 -6.86,-2.96 -6.86,-7.51 0,-4.55 3.03,-7.58 7.08,-7.58 4.05,0 6.86,2.96 6.86,7.51 -0.07,4.55 -3.11,7.58 -7.08,7.58 z m 0.07,-13.29 c -3.03,0 -4.98,2.38 -4.98,5.78 0,3.4 2.02,5.85 5.06,5.85 3.04,0 4.98,-2.38 4.98,-5.78 0,-3.4 -2.02,-5.85 -5.06,-5.85 z"
id="path49" />
<path
class="cls-17"
d="m 306.92,427.63 h -1.95 V 413.3 l 1.95,-0.43 v 3.03 c 0.94,-1.81 2.31,-3.03 4.19,-3.03 0.94,0 1.81,0.29 2.53,0.65 l -0.51,1.81 c -0.65,-0.36 -1.37,-0.65 -2.17,-0.65 -1.52,0 -2.89,1.16 -4.05,3.68 v 9.27 0 z"
id="path50" />
<path
class="cls-17"
d="m 321.25,427.99 c -4.33,0 -7.01,-3.11 -7.01,-7.51 0,-4.4 2.96,-7.58 7.01,-7.58 1.73,0 3.25,0.43 4.48,1.23 l -0.51,1.73 c -1.08,-0.72 -2.46,-1.16 -3.97,-1.16 -3.11,0 -4.98,2.31 -4.98,5.71 0,3.4 2.02,5.78 5.06,5.78 1.44,0 2.89,-0.43 3.97,-1.16 l 0.43,1.81 c -1.3,0.72 -2.82,1.16 -4.48,1.16 z"
id="path51" />
<path
class="cls-17"
d="m 337.67,427.63 v -9.27 c 0,-2.53 -1.01,-3.61 -3.03,-3.61 -1.66,0 -3.25,0.87 -4.41,2.02 v 10.86 h -1.95 v -21.05 l 1.95,-0.43 v 9.05 c 1.52,-1.52 3.4,-2.24 4.98,-2.24 2.75,0 4.48,1.81 4.48,4.91 v 9.78 h -2.02 z"
id="path52" />
</g>
</g>
<rect
class="cls-20"
x="576.57153"
y="48.263557"
width="183.05487"
height="59.208103"
rx="10.436842"
ry="10.436842"
id="rect53" />
<rect
class="cls-20"
x="576.57153"
y="137.89595"
width="183.05487"
height="59.208103"
rx="10.436842"
ry="10.436842"
id="rect54" />
<rect
class="cls-20"
x="576.57153"
y="227.52837"
width="183.05487"
height="59.208103"
rx="10.436842"
ry="10.436842"
id="rect55" />
<g
id="g57"
transform="matrix(0.9428042,0,0,0.9428042,-153.05524,-137.96856)">
<path
class="cls-17"
d="m 854.07,237.47 c -1.26,-0.73 -2.26,-1.73 -2.98,-3 -0.72,-1.27 -1.08,-2.69 -1.08,-4.25 0,-1.56 0.36,-2.98 1.08,-4.25 0.72,-1.27 1.71,-2.27 2.98,-3 1.27,-0.73 2.67,-1.09 4.21,-1.09 1.19,0 2.28,0.22 3.27,0.67 0.99,0.45 1.85,1.09 2.58,1.94 l -1.36,1.32 c -0.61,-0.73 -1.27,-1.26 -2,-1.61 -0.72,-0.34 -1.55,-0.51 -2.49,-0.51 -1.16,0 -2.22,0.27 -3.19,0.81 -0.97,0.54 -1.74,1.31 -2.31,2.3 -0.57,0.99 -0.86,2.13 -0.86,3.42 0,1.29 0.29,2.43 0.86,3.42 0.57,0.99 1.34,1.75 2.31,2.3 0.97,0.54 2.03,0.81 3.19,0.81 1.93,0 3.58,-0.79 4.93,-2.39 l 1.38,1.34 c -0.73,0.88 -1.64,1.57 -2.73,2.08 -1.09,0.51 -2.29,0.77 -3.58,0.77 -1.55,0 -2.95,-0.36 -4.21,-1.09 z"
id="path55" />
<path
class="cls-17"
d="m 867.32,222.24 h 5.37 c 0.89,0 1.72,0.2 2.48,0.6 0.76,0.4 1.36,0.96 1.82,1.68 0.45,0.72 0.68,1.54 0.68,2.44 0,0.9 -0.23,1.72 -0.68,2.44 -0.45,0.72 -1.06,1.28 -1.82,1.68 -0.76,0.4 -1.58,0.6 -2.48,0.6 h -3.48 v 6.51 h -1.9 v -15.97 z m 5.42,7.65 c 0.59,0 1.12,-0.14 1.57,-0.42 0.45,-0.28 0.81,-0.65 1.06,-1.09 0.25,-0.45 0.38,-0.91 0.38,-1.41 0,-0.5 -0.13,-0.96 -0.38,-1.41 -0.25,-0.45 -0.61,-0.81 -1.06,-1.09 -0.45,-0.28 -0.98,-0.42 -1.57,-0.42 h -3.52 v 5.84 z"
id="path56" />
<path
class="cls-17"
d="m 882.59,237.78 c -0.89,-0.53 -1.58,-1.26 -2.07,-2.21 -0.49,-0.94 -0.74,-2.03 -0.74,-3.27 v -10.06 h 1.9 v 10.15 c 0,1.29 0.34,2.35 1.03,3.16 0.68,0.81 1.67,1.22 2.94,1.22 1.27,0 2.26,-0.41 2.95,-1.22 0.69,-0.81 1.04,-1.86 1.04,-3.16 v -10.15 h 1.9 v 10.06 c 0,1.23 -0.23,2.33 -0.7,3.28 -0.47,0.95 -1.15,1.69 -2.04,2.21 -0.89,0.52 -1.94,0.78 -3.14,0.78 -1.2,0 -2.16,-0.26 -3.06,-0.79 z"
id="path57" />
</g>
<g
id="g60"
transform="matrix(0.9428042,0,0,0.9428042,-153.05524,-137.96856)">
<path
class="cls-17"
d="m 853.33,332.51 c -1.26,-0.73 -2.27,-1.73 -3.01,-3.01 -0.74,-1.28 -1.12,-2.69 -1.12,-4.24 0,-1.55 0.37,-2.96 1.12,-4.24 0.74,-1.28 1.75,-2.28 3.01,-3.01 1.26,-0.73 2.65,-1.09 4.15,-1.09 1.17,0 2.28,0.21 3.31,0.62 1.03,0.42 1.88,1 2.55,1.76 l -1.34,1.36 c -0.52,-0.62 -1.17,-1.1 -1.96,-1.44 -0.79,-0.33 -1.64,-0.5 -2.54,-0.5 -1.12,0 -2.16,0.27 -3.14,0.81 -0.98,0.54 -1.77,1.31 -2.35,2.3 -0.59,0.99 -0.88,2.13 -0.88,3.42 0,1.29 0.29,2.43 0.88,3.42 0.59,0.99 1.37,1.75 2.35,2.3 0.98,0.54 2.03,0.81 3.14,0.81 1.11,0 1.97,-0.17 2.69,-0.5 0.72,-0.33 1.34,-0.78 1.86,-1.33 0.39,-0.42 0.7,-0.92 0.94,-1.5 0.24,-0.58 0.39,-1.24 0.45,-1.95 h -5.91 v -1.76 h 7.67 c 0.07,0.42 0.11,0.8 0.11,1.16 0,0.98 -0.16,1.94 -0.47,2.87 -0.31,0.93 -0.81,1.74 -1.49,2.44 -1.47,1.59 -3.43,2.39 -5.87,2.39 -1.5,0 -2.88,-0.36 -4.15,-1.09 z"
id="path58" />
<path
class="cls-17"
d="m 868.12,317.28 h 5.37 c 0.89,0 1.72,0.2 2.48,0.6 0.76,0.4 1.36,0.96 1.82,1.68 0.45,0.72 0.68,1.54 0.68,2.44 0,0.9 -0.23,1.72 -0.68,2.44 -0.45,0.72 -1.06,1.28 -1.82,1.68 -0.76,0.4 -1.58,0.6 -2.48,0.6 h -3.48 v 6.51 h -1.9 v -15.97 z m 5.42,7.65 c 0.59,0 1.12,-0.14 1.57,-0.42 0.45,-0.28 0.81,-0.65 1.06,-1.09 0.25,-0.45 0.38,-0.91 0.38,-1.4 0,-0.49 -0.13,-0.96 -0.38,-1.41 -0.25,-0.45 -0.61,-0.81 -1.06,-1.09 -0.45,-0.28 -0.98,-0.42 -1.57,-0.42 h -3.52 v 5.84 h 3.52 z"
id="path59" />
<path
class="cls-17"
d="m 883.39,332.82 c -0.89,-0.53 -1.58,-1.26 -2.07,-2.21 -0.49,-0.94 -0.74,-2.03 -0.74,-3.27 v -10.06 h 1.9 v 10.15 c 0,1.29 0.34,2.35 1.03,3.16 0.68,0.81 1.67,1.22 2.94,1.22 1.27,0 2.26,-0.41 2.95,-1.22 0.69,-0.81 1.04,-1.86 1.04,-3.16 v -10.15 h 1.9 v 10.06 c 0,1.23 -0.23,2.33 -0.7,3.28 -0.47,0.95 -1.15,1.69 -2.04,2.21 -0.89,0.52 -1.94,0.78 -3.14,0.78 -1.2,0 -2.16,-0.26 -3.06,-0.79 z"
id="path60" />
</g>
<g
id="g63"
transform="matrix(0.9428042,0,0,0.9428042,-153.05524,-137.96856)">
<path
class="cls-17"
d="m 856.14,422.03 -3.93,-5.6 h 2.14 l 2.92,4.21 2.83,-4.21 h 2.23 l -3.99,5.6 4.06,5.78 h -2.21 l -2.94,-4.37 -2.92,4.37 h -2.21 l 4.01,-5.78 z"
id="path61" />
<path
class="cls-17"
d="m 864.48,411.84 h 5.37 c 0.89,0 1.72,0.2 2.48,0.6 0.76,0.4 1.36,0.96 1.82,1.68 0.45,0.72 0.68,1.54 0.68,2.44 0,0.9 -0.23,1.72 -0.68,2.44 -0.45,0.72 -1.06,1.28 -1.82,1.68 -0.76,0.4 -1.58,0.6 -2.48,0.6 h -3.48 v 6.51 h -1.9 v -15.97 z m 5.42,7.65 c 0.59,0 1.12,-0.14 1.57,-0.42 0.45,-0.28 0.81,-0.65 1.06,-1.09 0.25,-0.45 0.38,-0.92 0.38,-1.41 0,-0.49 -0.13,-0.96 -0.38,-1.41 -0.25,-0.45 -0.61,-0.81 -1.06,-1.09 -0.45,-0.28 -0.98,-0.42 -1.57,-0.42 h -3.52 v 5.84 z"
id="path62" />
<path
class="cls-17"
d="m 879.76,427.37 c -0.89,-0.53 -1.58,-1.26 -2.07,-2.21 -0.49,-0.94 -0.74,-2.03 -0.74,-3.27 v -10.06 h 1.9 v 10.15 c 0,1.29 0.34,2.35 1.03,3.16 0.68,0.81 1.67,1.22 2.94,1.22 1.27,0 2.26,-0.41 2.95,-1.22 0.69,-0.81 1.04,-1.86 1.04,-3.16 v -10.15 h 1.9 v 10.06 c 0,1.23 -0.23,2.33 -0.7,3.28 -0.47,0.95 -1.15,1.69 -2.04,2.21 -0.89,0.52 -1.94,0.78 -3.14,0.78 -1.2,0 -2.16,-0.26 -3.06,-0.79 z"
id="path63" />
</g>
<rect
class="cls-20"
x="313.99115"
y="115.41007"
width="152.01776"
height="49.16724"
rx="10.436842"
ry="10.436842"
id="rect63" />
<rect
class="cls-19"
x="280.23874"
y="10.739944"
width="219.51311"
height="313.52011"
rx="10.436842"
ry="10.436842"
id="rect64" />
<g
id="g73"
transform="matrix(0.9428042,0,0,0.9428042,-153.05524,-137.96856)">
<polygon
class="cls-17"
points="506.36,233.46 494.61,213.11 506.36,192.77 532.28,192.77 544.02,213.11 532.28,233.46 "
id="polygon64" />
<polygon
class="cls-14"
points="523.97,213.11 530.87,201.16 532.42,203.84 527.07,213.11 530.87,219.7 527.77,219.7 "
id="polygon65" />
<polygon
class="cls-15"
points="514.67,213.11 507.77,225.07 506.22,222.39 511.57,213.11 507.77,206.53 510.86,206.53 "
id="polygon66" />
<polygon
class="cls-16"
points="512.42,203.84 517.77,213.11 510.87,225.07 513.97,225.07 516.22,221.17 518.47,225.07 521.57,225.07 517.77,218.48 519.32,215.8 524.67,225.07 530.87,225.07 532.42,222.39 526.22,222.39 520.87,213.11 527.77,201.16 524.67,201.16 522.42,205.06 520.17,201.16 517.07,201.16 520.87,207.74 519.32,210.43 513.97,201.16 507.77,201.16 506.22,203.84 "
id="polygon67" />
<g
id="g70">
<path
class="cls-17"
d="m 556.38,213.11 c 0,-5.25 3.32,-8.6 8.17,-8.6 4.85,0 8.14,3.35 8.14,8.6 0,5.25 -3.3,8.6 -8.14,8.6 -4.84,0 -8.17,-3.4 -8.17,-8.6 z m 13.08,0 c 0,-3.56 -1.9,-5.65 -4.92,-5.65 -3.02,0 -4.92,2.09 -4.92,5.65 0,3.56 1.9,5.67 4.92,5.67 3.02,0 4.92,-2.18 4.92,-5.67 z"
id="path67" />
<path
class="cls-17"
d="m 587.03,215.13 c 0,4.27 -2.16,6.58 -5.27,6.58 -1.42,0 -2.54,-0.5 -3.3,-1.33 v 5.72 h -3.04 v -13.8 l -1.02,-2.07 2.94,-1.5 0.83,1.73 c 0.85,-1.02 2.16,-1.85 3.92,-1.85 3.06,0 4.94,2.49 4.94,6.51 z m -3.13,-0.02 c 0,-2.37 -1.07,-3.78 -3.13,-3.78 -0.83,0 -1.64,0.21 -2.3,0.59 v 6.39 c 0.64,0.43 1.42,0.66 2.23,0.66 1.99,0 3.21,-1.38 3.21,-3.87 z"
id="path68" />
<path
class="cls-17"
d="m 599.97,215.63 h -8.12 c 0.26,1.92 1.76,3.11 4.23,3.11 1.35,0 2.83,-0.29 3.8,-0.81 l -0.93,3.04 c -0.85,0.47 -2.14,0.74 -3.68,0.74 -4.13,0 -6.58,-2.59 -6.58,-6.58 0,-3.99 2.4,-6.51 6.29,-6.51 3.3,0 5.3,1.88 5.3,4.92 0,0.66 -0.09,1.5 -0.31,2.09 z m -8.05,-1.9 h 5.44 c -0.1,-1.83 -0.93,-2.61 -2.37,-2.61 -1.61,0 -2.71,0.97 -3.06,2.61 z"
id="path69" />
<path
class="cls-17"
d="m 613.46,212.66 v 8.74 h -3.04 v -7.53 c 0,-1.64 -0.66,-2.49 -2.23,-2.49 -0.9,0 -1.73,0.31 -2.42,0.62 v 9.4 h -3.04 v -9.12 l -1.02,-2.18 2.94,-1.4 0.95,1.95 c 1.38,-1.23 2.8,-1.99 4.25,-1.99 2.33,0 3.61,1.38 3.61,4.01 z"
id="path70" />
</g>
<path
class="cls-15"
d="m 629.35,204.83 -5.06,8.52 5.08,8.05 h -3.61 l -2.92,-4.7 -0.59,-1.38 h -0.14 l -0.55,1.35 -2.75,4.73 h -3.51 l 5.2,-8.38 -5.13,-8.19 h 3.56 l 3.06,4.92 0.5,1.26 h 0.14 l 0.55,-1.31 2.73,-4.87 z"
id="path71" />
<path
class="cls-15"
d="m 634.64,218.72 h 5.84 v 2.68 h -8.93 v -16.57 h 3.09 z"
id="path72" />
<path
class="cls-15"
d="m 652.67,221.4 -1.33,-4.23 h -5.72 l -1.35,4.23 h -3.13 l 5.41,-16.57 h 3.92 l 5.39,16.57 h -3.18 z m -6.24,-6.77 h 4.11 l -1.71,-5.41 -0.26,-1.28 h -0.17 l -0.26,1.28 z"
id="path73" />
</g>
<g
id="g82"
transform="matrix(0.9428042,0,0,0.9428042,-153.05524,-137.96856)">
<path
class="cls-17"
d="m 529.75,303.12 c -0.7,-0.31 -1.32,-0.77 -1.86,-1.37 -0.54,-0.6 -0.93,-1.33 -1.16,-2.19 l 1.68,-0.69 c 0.24,0.89 0.67,1.61 1.31,2.18 0.64,0.57 1.38,0.85 2.24,0.85 0.86,0 1.58,-0.23 2.18,-0.7 0.59,-0.46 0.89,-1.1 0.89,-1.9 0,-0.69 -0.25,-1.26 -0.76,-1.71 -0.51,-0.45 -1.33,-0.87 -2.48,-1.26 L 530.86,296 c -1.05,-0.37 -1.91,-0.86 -2.57,-1.47 -0.66,-0.61 -1,-1.45 -1,-2.53 0,-0.71 0.19,-1.36 0.58,-1.96 0.39,-0.6 0.93,-1.08 1.62,-1.44 0.69,-0.36 1.47,-0.54 2.33,-0.54 0.86,0 1.61,0.16 2.24,0.48 0.64,0.32 1.15,0.71 1.53,1.16 0.38,0.46 0.65,0.91 0.8,1.35 l -1.64,0.71 c -0.15,-0.51 -0.47,-0.98 -0.97,-1.39 -0.49,-0.42 -1.14,-0.62 -1.94,-0.62 -0.76,0 -1.41,0.21 -1.95,0.64 -0.54,0.43 -0.81,0.97 -0.81,1.62 0,0.58 0.23,1.06 0.69,1.42 0.46,0.37 1.15,0.71 2.08,1.03 l 0.98,0.33 c 1.26,0.46 2.24,1.03 2.95,1.71 0.71,0.68 1.06,1.62 1.06,2.79 0,0.96 -0.25,1.76 -0.74,2.41 -0.49,0.65 -1.11,1.13 -1.86,1.43 -0.75,0.3 -1.52,0.46 -2.3,0.46 -0.75,0 -1.47,-0.16 -2.17,-0.47 z"
id="path74" />
<path
class="cls-17"
d="m 541.44,303.38 c -0.37,-0.14 -0.67,-0.33 -0.92,-0.57 -0.55,-0.54 -0.83,-1.27 -0.83,-2.22 v -6.32 h -1.85 v -1.6 h 1.85 v -2.99 h 1.76 v 2.99 h 2.6 v 1.6 h -2.6 v 5.9 c 0,0.59 0.11,1.03 0.33,1.32 0.26,0.31 0.64,0.47 1.14,0.47 0.43,0 0.82,-0.12 1.16,-0.35 v 1.72 c -0.21,0.1 -0.42,0.17 -0.63,0.21 -0.21,0.04 -0.49,0.06 -0.82,0.06 -0.43,0 -0.83,-0.07 -1.19,-0.21 z"
id="path75" />
<path
class="cls-17"
d="m 547.27,303.13 c -0.61,-0.3 -1.08,-0.72 -1.41,-1.26 -0.33,-0.53 -0.5,-1.14 -0.5,-1.82 0,-1.12 0.42,-2 1.27,-2.63 0.84,-0.63 1.91,-0.95 3.2,-0.95 0.64,0 1.23,0.07 1.77,0.21 0.54,0.14 0.97,0.3 1.26,0.48 v -0.64 c 0,-0.79 -0.28,-1.42 -0.83,-1.9 -0.55,-0.48 -1.25,-0.72 -2.1,-0.72 -0.58,0 -1.12,0.13 -1.63,0.38 -0.51,0.26 -0.9,0.61 -1.19,1.07 l -1.33,-1 c 0.42,-0.64 0.99,-1.13 1.72,-1.5 0.73,-0.36 1.54,-0.54 2.43,-0.54 1.44,0 2.57,0.38 3.38,1.13 0.81,0.75 1.23,1.78 1.23,3.08 v 6.71 h -1.68 v -1.52 h -0.08 c -0.3,0.51 -0.76,0.95 -1.37,1.31 -0.61,0.36 -1.29,0.54 -2.06,0.54 -0.77,0 -1.47,-0.15 -2.08,-0.46 z m 3.9,-1.56 c 0.51,-0.3 0.92,-0.71 1.22,-1.22 0.3,-0.51 0.46,-1.07 0.46,-1.68 -0.33,-0.22 -0.74,-0.4 -1.23,-0.54 -0.49,-0.14 -1,-0.21 -1.54,-0.21 -0.97,0 -1.7,0.2 -2.19,0.6 -0.49,0.4 -0.74,0.92 -0.74,1.56 0,0.58 0.22,1.05 0.66,1.41 0.44,0.36 1,0.54 1.68,0.54 0.6,0 1.15,-0.15 1.66,-0.46 z"
id="path76" />
<path
class="cls-17"
d="m 560.19,303.04 c -0.64,-0.37 -1.12,-0.81 -1.44,-1.34 h -0.08 v 1.56 h -1.68 v -14.87 h 1.76 v 4.38 l -0.08,1.47 h 0.08 c 0.32,-0.54 0.8,-0.99 1.44,-1.36 0.64,-0.37 1.37,-0.55 2.17,-0.55 0.96,0 1.82,0.24 2.58,0.73 0.77,0.49 1.37,1.16 1.82,2.01 0.45,0.85 0.66,1.82 0.66,2.89 0,1.07 -0.22,2.04 -0.66,2.9 -0.44,0.85 -1.05,1.52 -1.82,2 -0.77,0.48 -1.63,0.73 -2.58,0.73 -0.8,0 -1.53,-0.18 -2.17,-0.55 z m 3.71,-1.55 c 0.54,-0.33 0.97,-0.8 1.29,-1.41 0.32,-0.61 0.48,-1.31 0.48,-2.12 0,-0.81 -0.16,-1.51 -0.48,-2.12 -0.32,-0.61 -0.75,-1.08 -1.29,-1.41 -0.54,-0.33 -1.11,-0.5 -1.72,-0.5 -0.61,0 -1.2,0.17 -1.73,0.5 -0.53,0.33 -0.96,0.8 -1.29,1.41 -0.33,0.61 -0.49,1.32 -0.49,2.12 0,0.8 0.16,1.51 0.49,2.12 0.33,0.61 0.75,1.08 1.29,1.41 0.53,0.33 1.11,0.5 1.73,0.5 0.62,0 1.18,-0.17 1.72,-0.5 z"
id="path77" />
<path
class="cls-17"
d="m 569.38,288.39 h 1.76 v 14.87 h -1.76 z"
id="path78" />
<path
class="cls-17"
d="m 575.65,302.86 c -0.8,-0.48 -1.43,-1.15 -1.88,-2 -0.45,-0.85 -0.67,-1.81 -0.67,-2.88 0,-1.01 0.21,-1.95 0.63,-2.81 0.42,-0.86 1.02,-1.55 1.81,-2.07 0.78,-0.51 1.68,-0.77 2.69,-0.77 1.01,0 1.94,0.23 2.7,0.7 0.76,0.46 1.35,1.1 1.75,1.92 0.4,0.82 0.61,1.75 0.61,2.8 0,0.17 -0.01,0.35 -0.04,0.54 h -8.39 c 0.04,0.8 0.24,1.48 0.58,2.03 0.35,0.55 0.79,0.97 1.32,1.25 0.53,0.28 1.09,0.42 1.67,0.42 1.38,0 2.43,-0.64 3.14,-1.91 l 1.49,0.73 c -0.44,0.84 -1.06,1.52 -1.85,2.02 -0.79,0.51 -1.74,0.76 -2.84,0.76 -1.01,0 -1.92,-0.24 -2.72,-0.73 z m 5.77,-6.02 c -0.03,-0.44 -0.15,-0.89 -0.37,-1.33 -0.22,-0.44 -0.57,-0.82 -1.04,-1.12 -0.47,-0.3 -1.07,-0.46 -1.79,-0.46 -0.83,0 -1.54,0.27 -2.12,0.8 -0.58,0.53 -0.96,1.24 -1.14,2.11 z"
id="path79" />
<path
class="cls-17"
d="m 585.7,288.39 h 1.76 v 6.42 h 7.64 v -6.42 h 1.76 v 14.87 h -1.76 v -6.77 h -7.64 v 6.77 h -1.76 z"
id="path80" />
<path
class="cls-17"
d="m 600.19,288.39 h 1.76 v 13.18 h 6.5 v 1.68 h -8.26 v -14.87 z"
id="path81" />
<path
class="cls-17"
d="m 613.46,302.55 c -1.16,-0.69 -2.08,-1.63 -2.74,-2.82 -0.66,-1.19 -1,-2.49 -1,-3.9 0,-1.41 0.33,-2.71 1,-3.9 0.66,-1.19 1.58,-2.13 2.74,-2.82 1.16,-0.69 2.46,-1.04 3.88,-1.04 1.42,0 2.72,0.35 3.88,1.04 1.16,0.69 2.08,1.63 2.74,2.82 0.66,1.19 1,2.49 1,3.9 0,1.41 -0.33,2.71 -1,3.9 -0.67,1.19 -1.58,2.13 -2.74,2.82 -1.16,0.69 -2.46,1.04 -3.88,1.04 -1.42,0 -2.72,-0.35 -3.88,-1.04 z m 6.82,-1.42 c 0.89,-0.52 1.6,-1.24 2.12,-2.16 0.52,-0.92 0.78,-1.97 0.78,-3.15 0,-1.18 -0.26,-2.22 -0.78,-3.15 -0.52,-0.93 -1.23,-1.64 -2.12,-2.16 -0.89,-0.52 -1.87,-0.78 -2.94,-0.78 -1.07,0 -2.05,0.26 -2.94,0.78 -0.89,0.52 -1.6,1.24 -2.12,2.16 -0.52,0.92 -0.78,1.97 -0.78,3.15 0,1.18 0.26,2.22 0.78,3.15 0.52,0.93 1.23,1.64 2.12,2.16 0.89,0.52 1.87,0.78 2.94,0.78 1.07,0 2.04,-0.26 2.94,-0.78 z"
id="path82" />
</g>
<rect
class="cls-20"
x="313.99115"
y="178.32341"
width="152.01776"
height="49.16724"
rx="10.436842"
ry="10.436842"
id="rect82" />
<g
id="g85"
transform="matrix(0.9428042,0,0,0.9428042,-153.05524,-137.96856)">
<path
class="cls-17"
d="m 562.61,362.19 -4.73,-7.18 H 560 l 3.74,5.79 h 0.08 l 3.74,-5.79 h 2.1 l -4.73,7.18 5.02,7.68 h -2.08 l -4.05,-6.21 h -0.08 l -4.07,6.21 h -2.08 z"
id="path83" />
<path
class="cls-17"
d="m 571.97,355.01 h 1.76 v 13.18 h 6.5 v 1.68 h -8.26 V 355 Z"
id="path84" />
<path
class="cls-17"
d="m 586.84,355.01 h 1.99 l 5.65,14.87 h -1.93 l -1.5,-4.09 h -6.44 l -1.5,4.09 h -1.93 l 5.65,-14.87 z m 3.61,9.12 -1.99,-5.36 -0.58,-1.6 h -0.08 l -0.58,1.6 -1.99,5.36 h 5.23 z"
id="path85" />
</g>
<rect
class="cls-20"
x="313.99115"
y="241.23671"
width="152.01776"
height="49.16724"
rx="10.436842"
ry="10.436842"
id="rect85" />
<g
id="g89"
transform="matrix(0.9428042,0,0,0.9428042,-153.05524,-137.96856)">
<path
class="cls-17"
d="m 555.86,421.85 h 5 c 0.83,0 1.6,0.19 2.3,0.56 0.71,0.37 1.27,0.9 1.69,1.57 0.42,0.67 0.63,1.43 0.63,2.27 0,0.84 -0.21,1.6 -0.63,2.27 -0.42,0.67 -0.99,1.19 -1.69,1.57 -0.71,0.37 -1.47,0.56 -2.3,0.56 h -3.24 v 6.06 h -1.76 v -14.87 z m 5.04,7.12 c 0.55,0 1.04,-0.13 1.46,-0.39 0.42,-0.26 0.75,-0.6 0.99,-1.02 0.24,-0.42 0.35,-0.85 0.35,-1.31 0,-0.46 -0.12,-0.89 -0.35,-1.31 -0.23,-0.42 -0.56,-0.75 -0.99,-1.02 -0.42,-0.26 -0.91,-0.39 -1.46,-0.39 h -3.28 v 5.44 z"
id="path86" />
<path
class="cls-17"
d="m 565.72,436.14 c -0.82,-0.6 -1.39,-1.45 -1.72,-2.54 l 1.66,-0.69 c 0.5,1.63 1.47,2.45 2.93,2.45 0.82,0 1.47,-0.28 1.96,-0.85 0.49,-0.57 0.74,-1.31 0.74,-2.24 v -10.42 h 1.76 v 10.28 c 0,1.04 -0.19,1.93 -0.57,2.67 -0.38,0.74 -0.91,1.3 -1.58,1.68 -0.67,0.38 -1.44,0.57 -2.31,0.57 -1.09,0 -2.05,-0.3 -2.87,-0.9 z"
id="path87" />
<path
class="cls-17"
d="m 576.13,421.85 h 5.02 c 0.82,0 1.58,0.19 2.28,0.56 0.71,0.37 1.27,0.9 1.69,1.57 0.42,0.67 0.63,1.43 0.63,2.27 0,0.69 -0.16,1.34 -0.47,1.93 -0.31,0.59 -0.74,1.09 -1.29,1.5 -0.55,0.4 -1.15,0.67 -1.82,0.81 l -0.04,0.06 4.19,6.08 v 0.08 h -2.08 l -4.05,-6.06 h -2.33 v 6.06 h -1.76 v -14.87 z m 4.92,7.16 c 0.51,0 0.99,-0.12 1.44,-0.35 0.45,-0.24 0.81,-0.56 1.08,-0.99 0.27,-0.42 0.4,-0.9 0.4,-1.42 0,-0.46 -0.12,-0.89 -0.35,-1.31 -0.23,-0.42 -0.56,-0.75 -0.99,-1.02 -0.42,-0.26 -0.91,-0.39 -1.46,-0.39 h -3.28 v 5.48 z"
id="path88" />
<path
class="cls-17"
d="m 591.34,423.53 h -4.17 v -1.68 h 10.09 v 1.68 h -4.15 v 13.18 h -1.77 z"
id="path89" />
</g>
<line
class="cls-21"
x1="203.42845"
y1="167.50002"
x2="280.23874"
y2="167.50002"
id="line89" />
<path
class="cls-21"
d="m 203.42847,77.867608 h 27.96357 c 5.76997,0 10.43685,4.676308 10.43685,10.436842 v 158.27797 c 0,5.76996 -4.67631,10.43684 -10.43685,10.43684 h -27.96357"
id="path90" />
<line
class="cls-21"
x1="576.57153"
y1="167.39632"
x2="499.76123"
y2="167.39632"
id="line90" />
<path
class="cls-21"
d="m 576.57151,257.02869 h -27.96356 c -5.76996,0 -10.43685,-4.67631 -10.43685,-10.43684 V 88.313878 c 0,-5.769962 4.67631,-10.436842 10.43685,-10.436842 h 27.96356"
id="path91" />
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 34 KiB

third_party/xla/docs/images/ops_2d_matrix.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 3.2 KiB

third_party/xla/docs/images/ops_alltoall.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 19 KiB

third_party/xla/docs/images/ops_alltoall_2.svg
Vendored
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 293 KiB

third_party/xla/docs/images/ops_concatenate.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 20 KiB

third_party/xla/docs/images/ops_convolution_1.svg
Vendored
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 498 KiB

third_party/xla/docs/images/ops_pad.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 59 KiB

third_party/xla/docs/images/ops_reduce_from_2d_matrix.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 11 KiB

third_party/xla/docs/images/ops_reduce_from_3d_matrix.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 8.3 KiB

third_party/xla/docs/images/ops_reduce_scatter_1.svg
Vendored
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 384 KiB

third_party/xla/docs/images/ops_reduce_window.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 3.2 KiB

third_party/xla/docs/images/ops_reduce_window_2.svg
Vendored
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 404 KiB

third_party/xla/docs/images/ops_reduce_window_stride.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 3.6 KiB

third_party/xla/docs/images/ops_scatter_1.svg
Vendored
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 528 KiB

third_party/xla/docs/images/ops_scatter_to_selected_window_element.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 59 KiB

third_party/xla/docs/images/ops_while.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 71 KiB

third_party/xla/docs/images/ops_xla_gather_0.svg
Vendored
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 88 KiB

third_party/xla/docs/images/ops_xla_gather_1.svg
Vendored
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 97 KiB

third_party/xla/docs/images/ops_xla_gather_2.svg
Vendored
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 118 KiB

third_party/xla/docs/images/partitioned_module.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 152 KiB

third_party/xla/docs/images/pre_layout_module.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 133 KiB

third_party/xla/docs/images/send_recv_order.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 14 KiB

third_party/xla/docs/images/send_recv_schedule.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 47 KiB

third_party/xla/docs/images/tf_xla_performance.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 22 KiB

third_party/xla/docs/images/triton_opt_pipeline.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 146 KiB

third_party/xla/docs/images/xla_array_layout_figure1.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 20 KiB

third_party/xla/docs/images/xla_array_layout_figure2.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 7.7 KiB

third_party/xla/docs/images/xla_hardware.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 71 KiB

third_party/xla/docs/index.md
Vendored
+38
View File
@@ -0,0 +1,38 @@
# XLA
XLA (Accelerated Linear Algebra) is an open-source compiler for machine
learning. The XLA compiler takes models from popular frameworks such as PyTorch,
TensorFlow, and JAX, and optimizes the models for high-performance execution
across different hardware platforms including GPUs, CPUs, and ML accelerators.
As a part of the OpenXLA project, XLA is built collaboratively by
industry-leading ML hardware and software companies, including
Alibaba, Amazon Web Services, AMD, Apple, Arm, Google, Intel, Meta, and NVIDIA.
## Key benefits
- **Build anywhere**: XLA is already integrated into leading ML frameworks
such as TensorFlow, PyTorch, and JAX.
- **Run anywhere**: It supports various backends including GPUs, CPUs, and ML
accelerators, and includes a pluggable infrastructure to add support for
more.
- **Maximize and scale performance**: It optimizes a model's performance with
production-tested optimization passes and automated partitioning for model
parallelism.
- **Eliminate complexity**: It leverages the power of
[MLIR](https://mlir.llvm.org/) to bring the best capabilities into a single
compiler toolchain, so you don't have to manage a range of domain-specific
compilers.
- **Future ready**: As an open source project, built through a collaboration
of leading ML hardware and software vendors, XLA is designed to operate at
the cutting-edge of the ML industry.
## Documentation
To learn more about XLA, check out the links on the left. If you're a new XLA
developer, you might want to start with [XLA architecture](architecture.md) and
then read [Contributing](contributing.md).
third_party/xla/docs/indexing.md
Vendored
+1032
View File
File diff suppressed because it is too large Load Diff
third_party/xla/docs/lhs_cost_model.md
Vendored
+235
View File
@@ -0,0 +1,235 @@
# LHS Cost Model
---
## tldr;
This page describes the internals of the cost model used by Latency Hiding
Scheduler. If you are interested in tuning the model go straight to the
[Tuning section](#tuning).
The Latency Hiding Scheduler (LHS) is a compiler pass that schedules a HLO DAG
in a way that minimizes wall time.
Its decisions are guided by the unified cost model, which uses a mixture of
performance tables and analytical models. In particular XLA embeds performance
tables for a GEMMs and fast-interconnect collectives, and uses analytical
networking and fusion cost model for other cases. The rest of the document
describes the inner workings of these on a high level.
---
## Performance tables – ICI collectives
Performance table consist of two main components: a collector and an
interpolator.
### Collector
[The **collector**](https://github.com/openxla/xla/blob/e6a0a911eb79f540d501458f953393ede9e0048c/xla/tools/collective_perf_table_gen_main.cc#L1) is a C++ tool responsible for generating the performance
tables for collective operations. It measures the performance of individual HLO
ops (e.g., `all-gather`, `all-reduce`) across a statically defined parameter
space.
#### How It Works
The tool performs a sweep over a range of collective ops, transfer sizes, and
transfer schemes for a given cluster. It uses the existing multi-host HLO runner
infrastructure and `ExecutionProfile` data to run the generated HLO and gather
performance metrics.
#### Data Collection Parameters
Latency tables are collected for a cross-product of the following parameters:
* **Collective Type**:
* `all-reduce`
* `all-gather`
* `reduce-scatter`
* **Transfer Size**:
* Logarithmic scale from 1024B up to 2GiB (e.g., 1024B, 2048B, 4096B, ...)
* **Transfer Scheme**:
* `rail-aligned`
* `non-rail-aligned`
This sweep is run for intra-node clusters with **2, 4, and 8 devices**.
#### Output
The result of a collection run is a latency table in `.pbtxt` format
(approximately 116 KB per platform).
### Interpolator
[The **interpolator**](https://github.com/openxla/xla/blob/e6a0a911eb79f540d501458f953393ede9e0048c/xla/service/gpu/model/collective_interpolator.h#L40) is the compiler component that consumes the generated
performance tables to provide runtime estimates during compilation.
#### Internal Data Structure
On initialization, the Interpolator processes the performance table into a map.
This map uses a tuple of `(collective_type, transfer_scheme)` as its **key**.
The **value** associated with each key is a 2D Euclidean plane. This plane
indexes the **network throughput** (measured by the Collector) based on two
axes:
1. Transfer size.
2. Number of devices involved.
#### Lookup and Interpolation
When the compiler encounters a collective operation, the Interpolator performs
the following steps:
1. It identifies the correct 2D throughput plane using the operation's `(collective_type, transfer_scheme)` as the map key.
2. It then uses a **weighted average retrieval** (based on Euclidean distance) within that 2D plane, using the operation's `(transfer_size, num_devices)` as the query point.
3. The result of this lookup is a single, unique **network throughput** value.
### Rationale: Throughput and Extrapolation
The system is designed to store **network throughput** rather than raw latency.
This design choice significantly simplifies extrapolating performance for
transfer sizes not explicitly present in the table.
If the latency tables capture network bandwidth saturation at a collective size
`S`, the throughput `T` at that point is considered the maximum. For any new
collective of size `S'` > `S`, the runtime can be estimated as:
<!-- mdformat off(disable mdformat for proper MathJax formatting) -->
$$\text{EstimatedTime}(S') = \frac{S'}{T_{\text{saturated}}}$$
<!-- mdformat on -->
This allows the model to estimate performance for collectives of any size, even
those larger than the 2GiB maximum measured by the Collector.
**Important:** This extrapolation model relies on the assumption that the
generated latency tables **capture true network bandwidth saturation**.
If the tables do not contain measurements at or beyond the saturation point, the
interpolator will:
* **Underestimate** the maximum throughput.
* Consequently, **overestimate** the runtime for large transfers.
In general XLA:GPU teams maintains performance tables, but in cases user decide
to provide their own, it is the responsibility of the user generating the tables
to ensure they are representative and include measurements in the
bandwidth-saturated region for the target hardware.
---
## Performance tables – GEMMs
Similar to the system for collectives, GEMM latency tables are supported by two
components: a **collector** and an **interpolator**.
### Collector
[The **collector**](https://github.com/openxla/xla/blob/e6a0a911eb79f540d501458f953393ede9e0048c/xla/tools/matmul_perf_table_gen_main.cc) is a C++ tool that computes performance tables for General
Matrix Multiplications (GEMMs). It measures the performance of matrix
multiplications at the HLO `dot` op level.
#### How It Works
The tool performs a sweep over a static space of GEMM dimensions (batch,
two non-contracting, and one contracting dimension) and data types.
* **Default Data Types:** `LHS = bf16,f32`, `RHS = bf16,f32`, `OUT = bf16,f32`.
* **Infrastructure:** Re-uses the HLO op profiler.
#### Collection Parameters
Latency tables are collected for a cross-product of the following dimensions:
* **batch:** `{1, 2, 4}`
* **m (non-contracting):** `{256, 512, ..., 4096}`
* **n (non-contracting):** `{256, 512, ..., 4096}`
* **k (contracting):** `{256, 512, ..., 4096}`
#### Output and Storage
A full sweep generates a `.pbtxt` latency table, ready to be consumed by
interpolator.
### Interpolator
[The **interpolator**](https://github.com/openxla/xla/blob/e6a0a911eb79f540d501458f953393ede9e0048c/xla/service/gpu/model/matmul_interpolator.h#L35) is the compiler component that uses the generated tables to estimate GEMM performance.
#### Rationale: FLOPS Saturation
The collected latency tables allow the interpolator to reconstruct **FLOPS** for
each entry:
<!-- mdformat off(disable mdformat for proper MathJax formatting) -->
$$\text{FLOPS} = \frac{2 \times b \times m \times n \times k}{\text{runtime}}$$
<!-- mdformat on -->
A key insight is that FLOPS **saturate** at a certain point; that is, the
hardware reaches peak FLOPS beyond a certain matrix shape. This saturation
allows the use of the same extrapolation method employed for collectives.
#### Lookup and Interpolation
The interpolator builds a **4D Euclidean space** from the table data. To provide
a performance estimate, it performs a **weighted-average interpolation** within
this 4D space. If there's no table for a certain data type, as a heuristic each
dimension is normalized to the number of bytes.
---
## Analytical Cost Model - DCN
### S-curve Collective Cost Model
The **S-curve** model is a fully analytical networking roofline model.
#### Overview
The model is designed to estimate the performance of collective operations based
on a set of fixed network properties.
#### Model Inputs
The model requires two categories of inputs:
1. **Fixed Network Properties (User-Defined):**
* Collective launch overhead
* NIC speed
* RTT (round trip time)
By default, XLA auto-detects a platform and uses values for the most common
architectures. These properties are configurable by the user. See
[Tuning section](#tuning) for details.
2. **Per-Collective Inputs:**
* Collective type (e.g., `AllGather`, `ReduceScatter`)
* Transfer size
* Number of nodes involved in the communication
#### Integration
The S-curve model is integrated into `XLA:GPU` and is being used on Hopper, and
Blackwell.
---
## Analytical Cost Model - Fusions
For other kernels we rely on the [GPU performance cost model](https://github.com/openxla/xla/blob/e6a0a911eb79f540d501458f953393ede9e0048c/xla/service/gpu/model/gpu_performance_model.h) to estimate the
right runtimes. You can read more about it [here](https://github.com/openxla/xla/discussions/10065).
---
## Tuning
S-curve model can be tuned by issuing right XLA flags. Default configuration
should be good enough in majority of cases, but the model control is exposed in
other cases.
```
export NIC_SPEED_GBPS=... # NIC speed per GPU in Gigabytes
export GPUS_PER_NODE=... # Num of GPUs per cluster interconnected with fast network (e.g. NVLINK)
export XLA_FLAGS=--xla_gpu_analytical_latency_estimator_options="nic_speed_gbps=$NIC_SPEED_GBPS,gpus_per_node=$GPUS_PER_NODE"
```
third_party/xla/docs/megascale/_toc.yaml
Vendored
+23
View File
@@ -0,0 +1,23 @@
# Copyright 2026 The OpenXLA Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================
toc:
- heading: Megascale debugging playbook
- title: Overview
path: xla/megascale/overview
- title: Getting started
section:
- title: Debugging Megascale
path: xla/megascale/debugging_workflow
third_party/xla/docs/megascale/debugging_workflow.md
Vendored
+336
View File
@@ -0,0 +1,336 @@
# Debugging Workflow
This document describe a general workflow for debugging MXLA issues.
## Prerequisite
1. Use JAX 0.6 or up, and enable JAX distributed service. This version of JAX
contains additional logging that can help identify which workers are
experiencing issues.
2. (Optional) Generate an HLO dump using the --xla_dump_to flag when
initializing your workload. This is discussed in the [XLA
documentation](https://openxla.org/xla/hlo_dumps).
3. (Optional) Set --vmodule=real_program_continuator=1 to enable verbose
logging for the TPU program execution status.
## Flow chart
The flowchart below illustrates the debugging process. To access detailed
playbooks for each step, click on the corresponding item in the chart.
<iframe src="flow_chart.svg" width="1000" height="2000" style="border: none;"></iframe>
<!-- linter style off -->
## Hangs
### Locate the Megascale Hang Detected Error
If you see the following error message in your TPU worker logs, this means that
MXLA timed out after detecting no progress:
```text
Megascale hang detected: Timed out waiting for 4 graphs to complete at launch_id 13650. Already completed: 100. StepGloballyInProgress: true. Timeout: 1m
```
1. Workers will report errors to a coordinator for processing.
* For **Pathways** jobs: the digest can be found in the logs of
`resource_manager` job.
* For **McJAX** jobs: the logs can be found on MXLA Coordinator. This is
typically task 0 of slice 0.
2. Check logs around time of error detection, and look for `Megascale detects a
hang`.
3. Follow steps below to diagnose the issue based on the identified cause.
### Diagnosis
#### Interpreting TPU States
Before diagnosing MXLA hangs, it is important to understand the TPU states
report format. Below is a sample report:
```text
Full error digest:
Potential cause: <determined_cause>
Potential culprit workers: <task_name>
First error timestamp: <timestamp>
First error type: <error_type>
TPU states:
Launch ID: <launch_id>
Module: jit.step_fn Fingerprint: <fingerprint>
Sample worker: <task_name>@<host_name>:<tpu_chip>:<tpu_core>
Tag:PC breakdown:
<num_cores>@<location>(HLO): [<task_name>:<host_name>@<tpu_chip>:<tpu_core>, ...]
...
```
#### Bad TPU Chip (tensor core or sparse core)
```text
Megascale detects a hang that is likely caused by bad TPU chips on the following hosts. Please remove the hosts from the fleet and restart the workload. If problem persists please contact Megascale XLA team.
The host that have bad TPUs are: <host_name>
Full error digest:
Potential cause: Bad TPU chips
Potential culprit workers: <job_name>/<task_id>:<host_name>
```
This error means that the issue is potentially caused by a faulty TPU chip. The
error message should include the job information and host name of the faulty
chip. In the example above, the faulty chip is on host `<host_name>`, affecting
task `<task_id>` of the job `<job_name>`. You can configure your job to avoid
that host.
**Note:** There are some cases that the hang was caused by a XLA or custom
pallas kernel bug, but if you see the same host appearing multiple times (for
example more than 3 times) in multiple hang events, the TPU on that host is very
likely faulty.
#### Networking issue
```text
Megascale detects a hang that is likely caused by a networking issue. Please examine the underlying networking stack for the following hosts.
The hosts are: <host_name>
Full error digest:
Potential cause: Networking issue
Potential culprit workers: <host_name_1>, <host_name_2>
```
This error indicates that your job has encountered a failed network link. The
error message should include a single or pair of job name, task id, host name of
the faulty network link. In the example above, the faulty network link is
between host `host_name_1` and `host_name_2`. Sometimes RapidEye can further
localize the faulty host if a single host appears in multiple broken network
links.
#### Different modules
```text
Megascale detects a hang that is likely caused by running different modules on different devices. Please confirm that all workers is running the exact same program. It can also be caused by a hang in a subset of devices and the unaffected devices have moved on to the next program. Please inspect the digest below to further root cause the hang.
Example hosts that have different HLO modules: <host_name>
Full error digest:
Potential cause: Different module
Potential culprit workers: <host_name>
TPU stats:
<host_name>: <pc>
TPU states:
Module: jit_loss_and_grad
Fingerprint: <fingerprint>
Launch ID: 193
<tag>:<pc>(<hlo>): <host_name>
Module: jit_optimizer_apply
Fingerprint: <fingerprint>
Launch ID: 0
<tag>:<pc>(<hlo>): <host_name>
```
This error may indicate that a hang has occurred in a subset of workers, causing
those workers to be stuck at the current module while unaffected workers advance
to the next module. To identify the root cause, inspect the digest printed by
RapidEye in the logs.
The `TPU states` section of the logs shows which modules are running on which
workers. In the example above, workers are running different modules:
`jit_loss_and_grad` and `jit_optimizer_apply`.
#### Fingerprint mismatch for HLO module
```text
Megascale detects a hang that is likely caused by inconsistent HLO module compilation across workers. This is likely a bug in JAX tracing or XLA compiler. Please inspect the HLO dumps to confirm the root cause.
Example hosts that have different HLO fingerprints: <host_name>
Full error digest:
Potential cause: Fingerprint mismatch
Potential culprit workers: <host_name>
TPU stats:
Module: reduce.31
Fingerprint: <fingerprint_1>
Launch ID: 37
<tag>:<pc>(<hlo>): <host_name>
Module: reduce.31
Fingerprint: <fingerprint_2>
Launch ID: 40
<tag>:<pc>(<hlo>): <host_name>
```
This log message indicates the hang was likely caused by inconsistent HLO module
compilation across workers, possibly due to an issue with JAX tracing or the XLA
compiler. If you see this log, follow [these
steps](https://openxla.org/xla/hlo_dumps) to collect HLO dumps from the culprit
workers for further debugging.
#### Data input stall
**Note:** This error is not yet implemented.
```text
Megascale detects a hang that is likely caused by data input stall on the
following hosts. Please check the workers to make sure the data input pipeline
is working properly.
The host that have data input stalls are: <host_name>
```
This error means that all devices launched the same program, but that input data
was not provided to the program before the system timed out. To fix this issue,
confirm that:
1. The identified hosts can access the input datasource.
2. The identified hosts are properly loading/parsing the input datasource.
3. Confirm identified hosts are not throttled on reads to the input datasource.
#### Unrecoverable error
```text
Some workers have halted with an unrecoverable error:
<worker> : {some error}
Please inspect the error log of these workers:
<worker>
```
This error means that there was an issue that prevented the program from
properly executing and could not be recovered automatically. This error was
unable to be specifically categorized. Further information can be obtained from
checking the logs of the worker(s) mentioned in the error report.
If the error appears to be specific to the given machine (ex. failure to copy
data from TPU to host), then you can configure your job to avoid those hosts.
#### Unknown Error
```text
Megascale detects a hang but cannot determine the root cause. Please inspect the
full digest below.
```
This error means that there was an issue that prevented the program from
properly executing and could not be recovered automatically. This error was
unable to be specifically categorized and there is no further error information
available.
<!-- linter style on -->
## Performance
### Get an XProf session
Follow the instructions in the [XProf
documentation](https://openxla.org/xprof/capturing_profiles) to generate an
XProf trace for your problematic run.
### Check for shortage of mapped DMA buffers
The Megascale XLA runtime needs to register host memory before it can be used
for DMAs to and from the TPU. This happens early after the process starts. If
you see these registrations (`MapDmaBuffer` calls) at steady-state then it
indicates that something is wrong. Look for the presence of these calls in XProf
Trace Viewer. See the screenshot below for reference.
**Tip:** Search for the exact worker name, because there can be other workers
with similar or close names. You also search for the term “MapDmaBuffer” on the
page.
![Example xprof trace showing MapDmaBuffer
calls](./images/map_dma_buffer_example_trace.png)
If the issue is observed then try to increase the size of the premapped memory
region by increasing the value of `--megascale_grpc_premap_memory_bytes`,
restarting the job, then checking again.
### Check for memory copies during network transfers
Megascale XLA network transfers are zero-copy by design. However, there are
cases where memory copies will occur and cause degraded performance. Look for
memory copies in Megascale's "Communication Transport" traces as shown in the
example screenshot below.
![Example "Megascale: Memory Copy" trace during network
receive](./images/memory_copy_example_trace.png)
If the issue is observed then try to increase the size of the premapped memory
region by increasing the value of `--megascale_grpc_premap_memory_bytes`,
restarting the job, then checking again.
### Network Analysis
MegaScale also provides a Colab
[notebook](https://github.com/openxla/xla/blob/main/xla/megascale/tools/network_analysis_oss.ipynb)
to help analyze network performance using an XProf trace.
This tool can be used to do the following:
* Examine transfer latencies to identify potential network slowdowns or host
slowness.
* Examine transfer sizes to identify if your workload is optimized to use a
smaller number of larger transfers as opposed to a large number of small
transfers.
* Determine if your workload is unoptimized and produces bursty
collectives/always has a large number of pending collectives.
* Visualize the network throughput timeline to see if the workload is network
bound.
* Examine {source, destination} pairs to identify possible bad hardware on
individual hosts.
#### Collective Slack Too Small
One indicator that your workload is not optimized for compute/communication
overlap is seeing small slack times for a subset of collectives. This can
manifest as longer than expected `recv-done` traces in the trace viewer, or as
collectives with zero or near-zero [slack
time](https://cloud.google.com/tpu/docs/troubleshooting/troubleshoot-multislice#slack_time).
If this is the case, look towards identifying bottlenecks in your workload that
may be causing parts of your program to not overlap compute and network
communication.
#### High Network Bandwidth Demand
If you are observing long `recv-done` op latencies within your model XProf, this
could be an indication that the model is 'Bandwidth Bound' in those portions of
the step function (is blocked by available network bandwidth in the system).
You can generate a timeline of network usage for your model. If you see
consistently high network usage throughout the step, or specific regions with
large spikes, then your model may be bandwidth bound in those regions.
Use the [Network Analysis](#network-analysis) to generate a timeline of network
usage: ![example](./images/bandwidth_bound.png)
To mitigate bandwidth bound models, you can:
1. Check the [Collective Slack](#collective-slack-too-small) of your model.
Models with many collectives with low slack will have bandwidth bound
regions.
2. Confirm that the network settings are optimized.
3. Examine your model structure and data sharding to see if there are ways to
increase computation/communication overlap.
4. (Data parallel models) Confirm that you have sufficient batch size in each
local replica to overlap with the communication.
#### High Network Latency
If the bandwidth is not saturated, you may want to generate the RPC latency
timeline. If you see high RPC latencies is constantly or sporadically high, this
means that there are some issues with the MXLA RPCs.
Use the [Network Analysis](#network-analysis) to generate the RPC latency
timeline, the following example shows that there is some sporadic 200ms long
tail RPC latencies. ![example](./images/tail_network_latency.png)
#### Confirm Optimal Network Settings
High RPC latencies on Cloud environment are often caused by suboptimal TCP
configuration. Please confirm if all TCP parameters are configured
properly within the container.
If any of the TCP parameters are not correctly configured, consult Google Cloud
ML Compute Services (CMCS) team on how to configure them properly.
#### HLO Dump
Please follow [these steps](https://openxla.org/xla/hlo_dumps) to dump HLO to
the local filesystem on the TPU worker. You may need to upload the dump to GCS
in order to share them with the XLA or Megascale team.
## Straggler
**Note on Future Tooling:** Google is actively working on open-sourcing versions
of diagnostic dashboards to provide a more streamlined experience for Cloud TPU
customers to identify and diagnose stragglers. These will be available soon.
third_party/xla/docs/megascale/flow_chart.svg
Vendored
+373
View File
@@ -0,0 +1,373 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<!-- Generated by graphviz version 2.44.1 (20201121.0304)
-->
<!-- Pages: 1 -->
<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" width="616pt" height="1440pt" viewBox="0.00 0.00 616.00 1440.00">
<g id="graph0" class="graph" transform="scale(0.904523 0.904523) rotate(0) translate(4 1588)">
<polygon fill="white" stroke="none" points="-4,4 -4,-1588 677,-1588 677,4 -4,4"/>
<!-- start -->
<g id="googleblue" class="node">
<title>start</title>
<g id="a_googleblue"><a xlink:title=" " target="_blank">
<path fill="none" stroke="black" d="M204,-1584C204,-1584 12,-1584 12,-1584 6,-1584 0,-1578 0,-1572 0,-1572 0,-1524 0,-1524 0,-1518 6,-1512 12,-1512 12,-1512 204,-1512 204,-1512 210,-1512 216,-1518 216,-1524 216,-1524 216,-1572 216,-1572 216,-1578 210,-1584 204,-1584"/>
<text xml:space="preserve" text-anchor="middle" x="108" y="-1532.88" font-family="Times,serif" font-size="40.00">☹️</text>
</a>
</g>
</g>
<!-- mxla_job -->
<g id="googlegreen" class="node">
<title>mxla_job</title>
<g id="a_googlegreen"><a xlink:title=" " target="_blank">
<polygon fill="lightgrey" stroke="black" points="108,-1476 0,-1440 108,-1404 216,-1440 108,-1476"/>
<text xml:space="preserve" text-anchor="middle" x="108" y="-1435.7" font-family="Times,serif" font-size="14.00">Is it a multi slice job?</text>
</a>
</g>
</g>
<!-- start&#45;&gt;mxla_job -->
<g id="edge1" class="edge">
<title>start-&gt;mxla_job</title>
<path fill="none" stroke="black" d="M108,-1511.7C108,-1504.1 108,-1495.94 108,-1487.97"/>
<polygon fill="black" stroke="black" points="111.5,-1488.02 108,-1478.02 104.5,-1488.02 111.5,-1488.02"/>
</g>
<!-- hang -->
<g id="googlegreen" class="node">
<title>hang</title>
<g id="a_googlegreen"><a xlink:title=" " target="_blank">
<polygon fill="lightgrey" stroke="black" points="229,-1368 121,-1332 229,-1296 337,-1332 229,-1368"/>
<text xml:space="preserve" text-anchor="middle" x="229" y="-1327.7" font-family="Times,serif" font-size="14.00">Does the job hang?</text>
</a>
</g>
</g>
<!-- mxla_job&#45;&gt;hang -->
<g id="edge3" class="edge">
<title>mxla_job-&gt;hang</title>
<path fill="none" stroke="black" d="M137.29,-1413.34C153.64,-1399.02 174.18,-1381.03 191.67,-1365.7"/>
<polygon fill="black" stroke="black" points="193.62,-1368.64 198.84,-1359.42 189.01,-1363.38 193.62,-1368.64"/>
<text xml:space="preserve" text-anchor="middle" x="158.39" y="-1390.36" font-family="Times,serif" font-size="14.00">Yes</text>
</g>
<!-- consult_xla -->
<g id="googlered" class="node">
<title>consult_xla</title>
<g id="a_googlered"><a xlink:title=" " target="_blank">
<path fill="none" stroke="black" d="M465,-288C465,-288 273,-288 273,-288 267,-288 261,-282 261,-276 261,-276 261,-228 261,-228 261,-222 267,-216 273,-216 273,-216 465,-216 465,-216 471,-216 477,-222 477,-228 477,-228 477,-276 477,-276 477,-282 471,-288 465,-288"/>
<text xml:space="preserve" text-anchor="middle" x="369" y="-247.7" font-family="Times,serif" font-size="14.00">Consult XLA team</text>
</a>
</g>
</g>
<!-- mxla_job&#45;&gt;consult_xla -->
<g id="edge2" class="edge">
<title>mxla_job-&gt;consult_xla</title>
<path fill="none" stroke="black" d="M92.06,-1409.12C71.96,-1368.6 40,-1293.42 40,-1225 40,-1225 40,-1225 40,-467 40,-402.89 14.87,-370.51 59,-324 116.66,-263.24 163.32,-304.88 249.72,-288.22"/>
<polygon fill="black" stroke="black" points="250.24,-291.69 259.27,-286.15 248.75,-284.85 250.24,-291.69"/>
<text xml:space="preserve" text-anchor="middle" x="31.75" y="-1134.95" font-family="Times,serif" font-size="14.00">No</text>
</g>
<!-- rapideye -->
<g id="googleyellow" class="node">
<title>rapideye</title>
<g id="a_googleyellow"><a xlink:href="debugging_workflow.md#hangs" xlink:title=" " target="_parent">
<path fill="none" stroke="black" d="M325,-1260C325,-1260 133,-1260 133,-1260 127,-1260 121,-1254 121,-1248 121,-1248 121,-1200 121,-1200 121,-1194 127,-1188 133,-1188 133,-1188 325,-1188 325,-1188 331,-1188 337,-1194 337,-1200 337,-1200 337,-1248 337,-1248 337,-1254 331,-1260 325,-1260"/>
<text xml:space="preserve" text-anchor="middle" x="229" y="-1219.7" font-family="Times,serif" font-size="14.00">Use RapidEye</text>
</a>
</g>
</g>
<!-- hang&#45;&gt;rapideye -->
<g id="edge4" class="edge">
<title>hang-&gt;rapideye</title>
<path fill="none" stroke="black" d="M229,-1295.7C229,-1287.93 229,-1279.57 229,-1271.43"/>
<polygon fill="black" stroke="black" points="232.5,-1271.66 229,-1261.66 225.5,-1271.66 232.5,-1271.66"/>
<text xml:space="preserve" text-anchor="middle" x="218.5" y="-1282.65" font-family="Times,serif" font-size="14.00">Yes</text>
</g>
<!-- check_straggler -->
<g id="googleyellow" class="node">
<title>check_straggler</title>
<g id="a_googleyellow"><a xlink:href="debugging_workflow.md#straggler" xlink:title=" " target="_parent">
<path fill="none" stroke="black" d="M585,-1260C585,-1260 393,-1260 393,-1260 387,-1260 381,-1254 381,-1248 381,-1248 381,-1200 381,-1200 381,-1194 387,-1188 393,-1188 393,-1188 585,-1188 585,-1188 591,-1188 597,-1194 597,-1200 597,-1200 597,-1248 597,-1248 597,-1254 591,-1260 585,-1260"/>
<text xml:space="preserve" text-anchor="middle" x="489" y="-1219.7" font-family="Times,serif" font-size="14.00">Check for straggler</text>
</a>
</g>
</g>
<!-- hang&#45;&gt;check_straggler -->
<g id="edge5" class="edge">
<title>hang-&gt;check_straggler</title>
<path fill="none" stroke="black" d="M276.69,-1311.56C308.85,-1298.45 352.43,-1280.68 391.74,-1264.65"/>
<polygon fill="black" stroke="black" points="392.83,-1267.99 400.77,-1260.97 390.19,-1261.51 392.83,-1267.99"/>
<text xml:space="preserve" text-anchor="middle" x="331.23" y="-1272.66" font-family="Times,serif" font-size="14.00">No</text>
</g>
<!-- has_straggler -->
<g id="googlegreen" class="node">
<title>has_straggler</title>
<g id="a_googlegreen"><a xlink:title=" " target="_blank">
<polygon fill="lightgrey" stroke="black" points="489,-1152 381,-1116 489,-1080 597,-1116 489,-1152"/>
<text xml:space="preserve" text-anchor="middle" x="489" y="-1111.7" font-family="Times,serif" font-size="14.00">Has straggler?</text>
</a>
</g>
</g>
<!-- check_straggler&#45;&gt;has_straggler -->
<g id="edge6" class="edge">
<title>check_straggler-&gt;has_straggler</title>
<path fill="none" stroke="black" d="M489,-1187.7C489,-1180.1 489,-1171.94 489,-1163.97"/>
<polygon fill="black" stroke="black" points="492.5,-1164.02 489,-1154.02 485.5,-1164.02 492.5,-1164.02"/>
</g>
<!-- outkast -->
<g id="googlered" class="node">
<title>outkast</title>
<g id="a_googlered"><a xlink:title=" " target="_blank">
<path fill="none" stroke="black" d="M612,-936C612,-936 420,-936 420,-936 414,-936 408,-930 408,-924 408,-924 408,-876 408,-876 408,-870 414,-864 420,-864 420,-864 612,-864 612,-864 618,-864 624,-870 624,-876 624,-876 624,-924 624,-924 624,-930 618,-936 612,-936"/>
<text xml:space="preserve" text-anchor="middle" x="516" y="-895.7" font-family="Times,serif" font-size="14.00">Remove bad host</text>
</a>
</g>
</g>
<!-- has_straggler&#45;&gt;outkast -->
<g id="edge7" class="edge">
<title>has_straggler-&gt;outkast</title>
<path fill="none" stroke="black" d="M493.28,-1081.09C497.83,-1045.05 505.05,-987.81 510.13,-947.56"/>
<polygon fill="black" stroke="black" points="513.59,-948.03 511.37,-937.67 506.65,-947.16 513.59,-948.03"/>
<text xml:space="preserve" text-anchor="middle" x="512.92" y="-1013.33" font-family="Times,serif" font-size="14.00">Yes</text>
</g>
<!-- has_dma_map -->
<g id="googlegreen" class="node">
<title>has_dma_map</title>
<g id="a_googlegreen"><a xlink:href="debugging_workflow.md#check_for_shortage_of_mapped_dma_buffers" xlink:title=" " target="_parent">
<polygon fill="lightgrey" stroke="black" points="380,-1044 272,-1008 380,-972 488,-1008 380,-1044"/>
<text xml:space="preserve" text-anchor="middle" x="380" y="-1003.7" font-family="Times,serif" font-size="14.00">Has DMAMap in the trace?</text>
</a>
</g>
</g>
<!-- has_straggler&#45;&gt;has_dma_map -->
<g id="edge11" class="edge">
<title>has_straggler-&gt;has_dma_map</title>
<path fill="none" stroke="black" d="M462.06,-1088.8C447.91,-1075.04 430.4,-1058.01 415.19,-1043.22"/>
<polygon fill="black" stroke="black" points="417.71,-1040.79 408.1,-1036.32 412.83,-1045.8 417.71,-1040.79"/>
<text xml:space="preserve" text-anchor="middle" x="426.23" y="-1066.68" font-family="Times,serif" font-size="14.00">No</text>
</g>
<!-- problem_persist -->
<g id="googlegreen" class="node">
<title>problem_persist</title>
<g id="a_googlegreen"><a xlink:title=" " target="_blank">
<polygon fill="lightgrey" stroke="black" points="565,-180 457,-144 565,-108 673,-144 565,-180"/>
<text xml:space="preserve" text-anchor="middle" x="565" y="-139.7" font-family="Times,serif" font-size="14.00">Does problem persist?</text>
</a>
</g>
</g>
<!-- problem_persist&#45;&gt;check_straggler -->
<g id="edge9" class="edge">
<title>problem_persist-&gt;check_straggler</title>
<path fill="none" stroke="black" d="M584.96,-173.62C610.64,-213.31 652,-288.21 652,-359 652,-1009 652,-1009 652,-1009 652,-1075.76 647.37,-1099.6 606,-1152 597.29,-1163.04 586.35,-1172.76 574.68,-1181.2"/>
<polygon fill="black" stroke="black" points="572.73,-1178.3 566.45,-1186.83 576.68,-1184.08 572.73,-1178.3"/>
<text xml:space="preserve" text-anchor="middle" x="641.5" y="-739.49" font-family="Times,serif" font-size="14.00">Yes</text>
</g>
<!-- happy_user -->
<g id="googlered" class="node">
<title>happy_user</title>
<g id="a_googlered"><a xlink:title=" " target="_blank">
<path fill="none" stroke="black" d="M661,-72C661,-72 469,-72 469,-72 463,-72 457,-66 457,-60 457,-60 457,-12 457,-12 457,-6 463,0 469,0 469,0 661,0 661,0 667,0 673,-6 673,-12 673,-12 673,-60 673,-60 673,-66 667,-72 661,-72"/>
<text xml:space="preserve" text-anchor="middle" x="565" y="-16.75" font-family="Times,serif" font-size="40.00">😊</text>
</a>
</g>
</g>
<!-- problem_persist&#45;&gt;happy_user -->
<g id="edge10" class="edge">
<title>problem_persist-&gt;happy_user</title>
<path fill="none" stroke="black" d="M565,-107.7C565,-99.93 565,-91.57 565,-83.43"/>
<polygon fill="black" stroke="black" points="568.5,-83.66 565,-73.66 561.5,-83.66 568.5,-83.66"/>
<text xml:space="preserve" text-anchor="middle" x="556.75" y="-94.65" font-family="Times,serif" font-size="14.00">No</text>
</g>
<!-- check_network_config -->
<g id="googleyellow" class="node">
<title>check_network_config</title>
<g id="a_googleyellow"><a xlink:title=" " target="_blank">
<path fill="none" stroke="black" d="M272,-504C272,-504 80,-504 80,-504 74,-504 68,-498 68,-492 68,-492 68,-444 68,-444 68,-438 74,-432 80,-432 80,-432 272,-432 272,-432 278,-432 284,-438 284,-444 284,-444 284,-492 284,-492 284,-498 278,-504 272,-504"/>
<text xml:space="preserve" text-anchor="middle" x="176" y="-463.7" font-family="Times,serif" font-size="14.00">Check network config</text>
</a>
</g>
</g>
<!-- is_network_config_optimal -->
<g id="googlegreen" class="node">
<title>is_network_config_optimal</title>
<g id="a_googlegreen"><a xlink:title=" " target="_blank">
<polygon fill="lightgrey" stroke="black" points="176,-396 68,-360 176,-324 284,-360 176,-396"/>
<text xml:space="preserve" text-anchor="middle" x="176" y="-355.7" font-family="Times,serif" font-size="14.00">is_network_config_optimal</text>
</a>
</g>
</g>
<!-- check_network_config&#45;&gt;is_network_config_optimal -->
<g id="edge23" class="edge">
<title>check_network_config-&gt;is_network_config_optimal</title>
<path fill="none" stroke="black" d="M176,-431.7C176,-424.1 176,-415.94 176,-407.97"/>
<polygon fill="black" stroke="black" points="179.5,-408.02 176,-398.02 172.5,-408.02 179.5,-408.02"/>
</g>
<!-- outkast&#45;&gt;problem_persist -->
<g id="edge8" class="edge">
<title>outkast-&gt;problem_persist</title>
<path fill="none" stroke="black" d="M545.23,-863.65C552.82,-852.81 560.2,-840.44 565,-828 588.07,-768.18 584,-749.11 584,-685 584,-685 584,-685 584,-359 584,-300.4 576.85,-233.14 571.32,-189.91"/>
<polygon fill="black" stroke="black" points="574.81,-189.61 570.04,-180.15 567.87,-190.52 574.81,-189.61"/>
</g>
<!-- increase_premap_buffer_size -->
<g id="googleyellow" class="node">
<title>increase_premap_buffer_size</title>
<g id="a_googleyellow"><a xlink:title=" " target="_blank">
<path fill="none" stroke="black" d="M544,-828C544,-828 352,-828 352,-828 346,-828 340,-822 340,-816 340,-816 340,-768 340,-768 340,-762 346,-756 352,-756 352,-756 544,-756 544,-756 550,-756 556,-762 556,-768 556,-768 556,-816 556,-816 556,-822 550,-828 544,-828"/>
<text xml:space="preserve" text-anchor="middle" x="448" y="-787.7" font-family="Times,serif" font-size="14.00">Increase premap buffer size</text>
</a>
</g>
</g>
<!-- increase_premap_buffer_size&#45;&gt;problem_persist -->
<g id="edge15" class="edge">
<title>increase_premap_buffer_size-&gt;problem_persist</title>
<path fill="none" stroke="black" d="M475.56,-755.77C504.55,-715.1 546,-645.04 546,-577 546,-577 546,-577 546,-359 546,-300.4 553.15,-233.14 558.68,-189.91"/>
<polygon fill="black" stroke="black" points="562.13,-190.52 559.96,-180.15 555.19,-189.61 562.13,-190.52"/>
</g>
<!-- get_hlo_dump -->
<g id="googleyellow" class="node">
<title>get_hlo_dump</title>
<g id="a_googleyellow"><a xlink:href="debugging_workflow.md#hlo_dump" xlink:title=" " target="_parent">
<path fill="none" stroke="black" d="M506,-612C506,-612 314,-612 314,-612 308,-612 302,-606 302,-600 302,-600 302,-552 302,-552 302,-546 308,-540 314,-540 314,-540 506,-540 506,-540 512,-540 518,-546 518,-552 518,-552 518,-600 518,-600 518,-606 512,-612 506,-612"/>
<text xml:space="preserve" text-anchor="middle" x="410" y="-571.7" font-family="Times,serif" font-size="14.00">Get HLO Dump</text>
</a>
</g>
</g>
<!-- get_hlo_dump&#45;&gt;consult_xla -->
<g id="edge20" class="edge">
<title>get_hlo_dump-&gt;consult_xla</title>
<path fill="none" stroke="black" d="M405.5,-539.62C398.06,-481.21 383.21,-364.61 374.93,-299.57"/>
<polygon fill="black" stroke="black" points="378.44,-299.42 373.7,-289.94 371.49,-300.3 378.44,-299.42"/>
</g>
<!-- update_network_config -->
<g id="googleryellow" class="node">
<title>update_network_config</title>
<g id="a_googleryellow"><a xlink:title=" " target="_blank">
<path fill="none" stroke="black" d="M231,-288C231,-288 39,-288 39,-288 33,-288 27,-282 27,-276 27,-276 27,-228 27,-228 27,-222 33,-216 39,-216 39,-216 231,-216 231,-216 237,-216 243,-222 243,-228 243,-228 243,-276 243,-276 243,-282 237,-288 231,-288"/>
<text xml:space="preserve" text-anchor="middle" x="135" y="-247.7" font-family="Times,serif" font-size="14.00">Update network config</text>
</a>
</g>
</g>
<!-- update_network_config&#45;&gt;problem_persist -->
<g id="edge24" class="edge">
<title>update_network_config-&gt;problem_persist</title>
<path fill="none" stroke="black" d="M243.29,-218.32C246.22,-217.53 249.13,-216.75 252,-216 332.07,-194.97 424.63,-174.43 488.21,-160.9"/>
<polygon fill="black" stroke="black" points="488.81,-164.35 497.86,-158.86 487.35,-157.51 488.81,-164.35"/>
</g>
<!-- has_dma_map&#45;&gt;increase_premap_buffer_size -->
<g id="edge12" class="edge">
<title>has_dma_map-&gt;increase_premap_buffer_size</title>
<path fill="none" stroke="black" d="M380.12,-971.78C381.28,-942.12 385.48,-899.05 399,-864 402.45,-855.06 407.23,-846.16 412.47,-837.83"/>
<polygon fill="black" stroke="black" points="415.31,-839.87 417.93,-829.61 409.48,-836 415.31,-839.87"/>
<text xml:space="preserve" text-anchor="middle" x="378.56" y="-902.37" font-family="Times,serif" font-size="14.00">Yes</text>
</g>
<!-- has_memcpy -->
<g id="googlegreen" class="node">
<title>has_memcpy</title>
<g id="a_googlegreen"><a xlink:href="debugging_workflow.md#check_for_memory_copies_during_network_transfers" xlink:title=" " target="_parent">
<polygon fill="lightgrey" stroke="black" points="244,-936 136,-900 244,-864 352,-900 244,-936"/>
<text xml:space="preserve" text-anchor="middle" x="244" y="-895.7" font-family="Times,serif" font-size="14.00">Has 'Memory Copy' in the trace?</text>
</a>
</g>
</g>
<!-- has_dma_map&#45;&gt;has_memcpy -->
<g id="edge13" class="edge">
<title>has_dma_map-&gt;has_memcpy</title>
<path fill="none" stroke="black" d="M348.11,-982.15C329.19,-967.4 305,-948.54 284.7,-932.73"/>
<polygon fill="black" stroke="black" points="286.92,-930.01 276.88,-926.63 282.62,-935.54 286.92,-930.01"/>
<text xml:space="preserve" text-anchor="middle" x="303.53" y="-958.53" font-family="Times,serif" font-size="14.00">No</text>
</g>
<!-- has_memcpy&#45;&gt;increase_premap_buffer_size -->
<g id="edge14" class="edge">
<title>has_memcpy-&gt;increase_premap_buffer_size</title>
<path fill="none" stroke="black" d="M285.28,-877.55C309.55,-864.94 341.02,-848.59 369.82,-833.63"/>
<polygon fill="black" stroke="black" points="371.27,-836.81 378.53,-829.1 368.04,-830.6 371.27,-836.81"/>
<text xml:space="preserve" text-anchor="middle" x="322.19" y="-839.62" font-family="Times,serif" font-size="14.00">Yes</text>
</g>
<!-- no_slack_collectives -->
<g id="googlegreen" class="node">
<title>no_slack_collectives</title>
<g id="a_googlegreen"><a xlink:href="debugging_workflow.md#collective_slack_too_small" xlink:title=" " target="_parent">
<polygon fill="lightgrey" stroke="black" points="214,-828 106,-792 214,-756 322,-792 214,-828"/>
<text xml:space="preserve" text-anchor="middle" x="214" y="-787.7" font-family="Times,serif" font-size="14.00">Collective with small slack?</text>
</a>
</g>
</g>
<!-- has_memcpy&#45;&gt;no_slack_collectives -->
<g id="edge16" class="edge">
<title>has_memcpy-&gt;no_slack_collectives</title>
<path fill="none" stroke="black" d="M234.84,-866.62C232.1,-856.96 229.06,-846.22 226.17,-836"/>
<polygon fill="black" stroke="black" points="229.61,-835.31 223.52,-826.64 222.88,-837.21 229.61,-835.31"/>
<text xml:space="preserve" text-anchor="middle" x="220.8" y="-850.87" font-family="Times,serif" font-size="14.00">No</text>
</g>
<!-- high_bandwidth_demand -->
<g id="googlegreen" class="node">
<title>high_bandwidth_demand</title>
<g id="a_googlegreen"><a xlink:href="debugging_workflow.md#high_network_bandwidth_demand" xlink:title=" " target="_parent">
<polygon fill="lightgrey" stroke="black" points="214,-720 106,-684 214,-648 322,-684 214,-720"/>
<text xml:space="preserve" text-anchor="middle" x="214" y="-679.7" font-family="Times,serif" font-size="14.00">High network bandwidth demand?</text>
</a>
</g>
</g>
<!-- high_bandwidth_demand&#45;&gt;get_hlo_dump -->
<g id="edge19" class="edge">
<title>high_bandwidth_demand-&gt;get_hlo_dump</title>
<path fill="none" stroke="black" d="M254.61,-661.04C277.68,-648.56 307.3,-632.54 334.48,-617.84"/>
<polygon fill="black" stroke="black" points="335.9,-621.05 343.04,-613.22 332.57,-614.89 335.9,-621.05"/>
<text xml:space="preserve" text-anchor="middle" x="288.95" y="-623.49" font-family="Times,serif" font-size="14.00">Yes</text>
</g>
<!-- high_network_latency -->
<g id="googlegreen" class="node">
<title>high_network_latency</title>
<g id="a_googlegreen"><a xlink:href="debugging_workflow.md#high_network_latency" xlink:title=" " target="_parent">
<polygon fill="lightgrey" stroke="black" points="176,-612 68,-576 176,-540 284,-576 176,-612"/>
<text xml:space="preserve" text-anchor="middle" x="176" y="-571.7" font-family="Times,serif" font-size="14.00">High network latency?</text>
</a>
</g>
</g>
<!-- high_bandwidth_demand&#45;&gt;high_network_latency -->
<g id="edge21" class="edge">
<title>high_bandwidth_demand-&gt;high_network_latency</title>
<path fill="none" stroke="black" d="M202.7,-651.48C199.06,-641.32 194.96,-629.89 191.09,-619.1"/>
<polygon fill="black" stroke="black" points="194.5,-618.24 187.83,-610 187.91,-620.6 194.5,-618.24"/>
<text xml:space="preserve" text-anchor="middle" x="186.7" y="-634.56" font-family="Times,serif" font-size="14.00">No</text>
</g>
<!-- high_network_latency&#45;&gt;check_network_config -->
<g id="edge22" class="edge">
<title>high_network_latency-&gt;check_network_config</title>
<path fill="none" stroke="black" d="M176,-539.7C176,-531.93 176,-523.57 176,-515.43"/>
<polygon fill="black" stroke="black" points="179.5,-515.66 176,-505.66 172.5,-515.66 179.5,-515.66"/>
<text xml:space="preserve" text-anchor="middle" x="165.5" y="-526.65" font-family="Times,serif" font-size="14.00">Yes</text>
</g>
<!-- high_network_latency&#45;&gt;consult_xla -->
<g id="edge25" class="edge">
<title>high_network_latency-&gt;consult_xla</title>
<path fill="none" stroke="black" d="M224.3,-555.69C248.05,-543.96 275.31,-526.81 293,-504 340.05,-443.33 358.09,-353.68 364.93,-299.56"/>
<polygon fill="black" stroke="black" points="368.39,-300.08 366.08,-289.74 361.44,-299.27 368.39,-300.08"/>
<text xml:space="preserve" text-anchor="middle" x="322.68" y="-441.05" font-family="Times,serif" font-size="14.00">No</text>
</g>
<!-- no_slack_collectives&#45;&gt;get_hlo_dump -->
<g id="edge17" class="edge">
<title>no_slack_collectives-&gt;get_hlo_dump</title>
<path fill="none" stroke="black" d="M259.25,-770.64C282.73,-758.44 310.68,-741.17 331,-720 357.97,-691.91 378.74,-652.81 392.2,-622.6"/>
<polygon fill="black" stroke="black" points="395.29,-624.27 396.05,-613.7 388.86,-621.49 395.29,-624.27"/>
<text xml:space="preserve" text-anchor="middle" x="333.85" y="-709.34" font-family="Times,serif" font-size="14.00">Yes</text>
</g>
<!-- no_slack_collectives&#45;&gt;high_bandwidth_demand -->
<g id="edge18" class="edge">
<title>no_slack_collectives-&gt;high_bandwidth_demand</title>
<path fill="none" stroke="black" d="M214,-755.7C214,-748.1 214,-739.94 214,-731.97"/>
<polygon fill="black" stroke="black" points="217.5,-732.02 214,-722.02 210.5,-732.02 217.5,-732.02"/>
<text xml:space="preserve" text-anchor="middle" x="205.75" y="-743.05" font-family="Times,serif" font-size="14.00">No</text>
</g>
<!-- is_network_config_optimal&#45;&gt;consult_xla -->
<g id="edge27" class="edge">
<title>is_network_config_optimal-&gt;consult_xla</title>
<path fill="none" stroke="black" d="M215.99,-337.04C238.61,-324.62 267.62,-308.68 294.28,-294.04"/>
<polygon fill="black" stroke="black" points="295.96,-297.11 303.04,-289.22 292.59,-290.97 295.96,-297.11"/>
<text xml:space="preserve" text-anchor="middle" x="249.75" y="-299.43" font-family="Times,serif" font-size="14.00">Yes</text>
</g>
<!-- is_network_config_optimal&#45;&gt;update_network_config -->
<g id="edge26" class="edge">
<title>is_network_config_optimal-&gt;update_network_config</title>
<path fill="none" stroke="black" d="M163.92,-327.77C160.45,-318.81 156.6,-308.84 152.86,-299.18"/>
<polygon fill="black" stroke="black" points="156.14,-297.95 149.27,-289.88 149.61,-300.47 156.14,-297.95"/>
<text xml:space="preserve" text-anchor="middle" x="147.96" y="-294.53" font-family="Times,serif" font-size="14.00">No</text>
</g>
</g>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 22 KiB

third_party/xla/docs/megascale/images/bandwidth_bound.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 79 KiB

third_party/xla/docs/megascale/images/map_dma_buffer_example_trace.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 145 KiB

third_party/xla/docs/megascale/images/megascale_stats_zero_slack_example.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 114 KiB

third_party/xla/docs/megascale/images/memory_copy_example_trace.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 118 KiB

third_party/xla/docs/megascale/images/tail_network_latency.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 248 KiB

third_party/xla/docs/megascale/overview.md
Vendored
+54
View File
@@ -0,0 +1,54 @@
<!-- linter style off -->
# Megascale XLA
MegascaleXLA is a compiler + runtime system that powers large-scale TPU
training. It implements collective communication primitives that allow multiple
TPU slices to communicate, which allows running training jobs that span beyond
the limits of a single ICI domain.
The [debugging guide](./debugging_workflow.md) discusses how to identify and
diagnose sources of performance issues such as slowness, hangs or errors in a
multi-slice job driven by Megascale.
## Terminology
- **Slice**
- A slice is a collection of chips all located inside the same TPU Pod
connected by high-speed inter chip interconnects (ICI). Slices are
described in terms of chips or TensorCores, depending on the TPU
version.
- Multislice is a group of slices, extending TPU connectivity beyond the
inter-chip interconnect (ICI) connections and leveraging the data-center
network (DCN) for transmitting data beyond a slice. Data within each slice
is still transmitted by ICI. Using this hybrid connectivity, Multislice
enables parallelism across slices and lets you use a greater number of TPU
cores for a single job than what a single slice can accommodate.
- TPUs can be used to run a job either on a single slice or multiple
slices.
- **RapidEye**
- RapidEye is a system that aims to provide global ML debugging
infrastructure to quickly identify and root cause issues caused by bad
hardware or software bugs. It monitors MegaScale jobs to automatically
detect, analyze, and classify hang events. The process involves
collecting data from all job workers, coordinating responses when a hang
occurs, and generating a summary digest file for each event.
- RapidEye is enabled by default for all multislice workloads. The
diagnosis can be found under the Pathways resource manager or MXLA
coordinator (slice0 task0 for multi-controller JAX workload). RapidEye
is also used for auto-removing bad TPUs and NICs based on fleetwide
rapideye data.
- **Megascale Collective**
- XLA collectives are supported via the Megascale XLA (MXLA) primitives
which are not directly usable by the end user. At the time of writing
this the MXLA primitives include collectives include AllGather,
AllReduce, AllToAll, ReduceScatter and OneToOne. The reduction
operations that are currently supported include summation, max/min and
product.
<!-- linter style on -->
third_party/xla/docs/oom_debugging.md
Vendored
+88
View File
@@ -0,0 +1,88 @@
# Debug OOM errors with XProf
Out Of Memory (OOM) errors occur when the accelerator's (GPU or TPU) High
Bandwidth Memory (HBM) capacity is exhausted. Some common causes for OOM issues
and debugging techniques are detailed in
[E1000 - Compile Time HBM OOM documentation](./errors/error_1000.md) and
[JAX documentation on GPU memory allocation](https://docs.jax.dev/en/latest/gpu_memory_allocation.html#common-causes-of-oom-failures).
This page describes how to use **XProf's Memory Viewer tool** to visualize your
JAX program's memory usage, identify peak usage instances, and debug OOM errors.
This involves the following steps:
1. Run your program with
[`jax.profiler.trace`](https://docs.jax.dev/en/latest/_autosummary/jax.profiler.trace.html#jax.profiler.trace)
to capture the profile.
2. Start XProf in the background, and use the
[Memory Viewer tool](https://openxla.org/xprof/memory_viewer) to view memory
utilization details.
## Example program
The following JAX program leads to an OOM error:
```python
import jax
from jax import random
import jax.numpy as jnp
@jax.profiler.trace("/tmp/xprof")
@jax.jit
def oom():
a = random.normal(random.PRNGKey(1), (327680, 327680), dtype=jnp.bfloat16)
return a @ a
if __name__ == "__main__":
oom()
```
**Note:** Prefer `jax.profiler.trace` instead of
`jax.profiler.start_trace`/`jax.profiler.stop_trace` because the
`jax.profiler.trace` context manager handles profiling in an exception safe
manner.
On a TPU-machine, this program fails with:
```shell
XlaRuntimeError: RESOURCE_EXHAUSTED: Allocation (size=107374182400) would exceed memory (size=17179869184) :: #allocation7 [shape = 'u8[327680,327680]{1,0:T(8,128)(4,1)}', space=hbm, size = 0xffffffffffffffff, tag = 'output of xor_convert_fusion@{}'] :: <no-hlo-instruction>
```
(On a GPU-machine, the error looks like: `XlaRuntimeError: RESOURCE_EXHAUSTED:
Out of memory while trying to allocate 214748364800 bytes.`)
## Run XProf
Install `xprof` (`pip install xprof`), and start an XProf instance specifying
the directory where the profile is stored:
```shell
xprof --logdir=/tmp/xprof/ --port=6006
```
Go to the instance (on a local machine, at `http://localhost:6006`). In the
*Tools* dropdown, select *Memory Viewer*, and in the Memory Viewer tool window,
select *HBM* in the *Memory Types* dropdown (usually selected by default).
![XProf Memory Viewer page for the above example program](images/oom_debugging_example_memory_viewer.png)
The
[XProf: Memory Viewer tool documentation](https://openxla.org/xprof/memory_viewer#memory_viewer_components)
describes the components of the tool and the information presented.
Focus on the *HLO Ops at Peak Memory Allocation* section that shows three buffer
charts at the peak memory usage point. The buffer includes:
- **Program Inputs and Outputs:** Training batches, optimizer states, etc.
- **TensorCore and SparseCore Temporaries:** Dynamic memory required for
intermediate calculations (like activations, gradients, etc.)
You can hover on the buffer charts to get more details about the Op like it's
size, shape, allocation type, and more. This can help you identify and evaluate
Ops that may have high or long-lasting temporaries, any large
input/intermediate/output tensors that have inefficient padding, etc., that are
contributing to the peak memory and need to be adjusted or optimized.
Learn specific debugging techniques in
[E1000: Debugging](https://openxla.org/xla/errors/error_1000#debugging).
third_party/xla/docs/operation_semantics.md
Vendored
+5716
View File
File diff suppressed because it is too large Load Diff
third_party/xla/docs/persisted_autotuning.md
Vendored
+108
View File
@@ -0,0 +1,108 @@
# Persisted autotuning (GPU only)
We use OpenAI Triton for generating some of the GPU kernels. Triton allows
generating fast GPU kernels for certain fusions, but we have to tune some
parameters for each such fusion.
This can take a long time if there are many fusions, so we provide a way to load
those autotuning results, while still running the other compilation steps
normally. Autotuning caches are still useful if we make a few changes: the
fusions that are present in the cache will use the cache, and the other ones
will be autotuned normally.
## Recommended: Cache directory
```
--xla_gpu_per_fusion_autotune_cache_dir=your/directory
```
Use and maintain a per-fusion autotune cache in the given directory. There will
be one file per distinct fusion.
The main advantage of this approach is that you can use the same cache directory
for multiple XLA runs (of different models) and your cache will grow with each
new fusion encountered - speeding up subsequent runs. There is also basic
support for running multiple XLA instances with the same cache directory
concurrently.
XLA will read existing results when they are needed and write new results after
they are determined.
- The directory must exist before running XLA and it must be writable.
- Cache invalidation has to be handled by the user:
- Please use an empty directory if you want to start with an empty cache.
- XLA version checks must be done by the user:
- If you want to use separate caches for different versions of XLA, please
use different directories.
The cache is turned off by default (when you don't provide the parameter).
Limitation: This is not guaranteed to work well in combination with the other
caching method described below.
## Alternative: Loading or dumping all results from a given HLO to one file
The autotuning results can be dumped/loaded using these parameters:
```
--xla_gpu_dump_autotune_results_to=
--xla_gpu_load_autotune_results_from=
```
If we specify a .txt or .textproto file, then the cache will be dumped in
textproto format, otherwise in binary protobuf format.
## In tests
Persisted autotuning can also be used in tests. It is recommended to use it if
the tests are very big, especially if the performance of the test environment is
limited.
It only works well if the autotune cache contains results generated on the same
type of GPU where the tests are being run.
### Making a test use persisted autotuning
For now let's assume that the test in question always uses the same GPU type.
1. We have to export the autotune results from the test, for example by
specifying these parameters to the test command:
```
--test_env=XLA_FLAGS=--xla_gpu_dump_autotune_results_to=TEST_UNDECLARED_OUTPUTS_DIR/autotune_cache.textproto
--test_sharding_strategy=disabled
```
Sharding must be disabled to correctly get a single autotune cache for all
tests.
2. Then we have to upload that cache to our code repository.
3. Then we have to add the cache to the data dependencies of our test target,
and load it using an environment variable.
```
data = ["test_autotune_cache.textproto"],
env = {"XLA_FLAGS": "--xla_gpu_load_autotune_results_from=" +
"$(execpath test_autotune_cache.textproto)"},
```
(It is OK to use sharding in tests that load autotune results.)
Please also see the example tests in
[xla/backends/gpu/tests/BUILD](https://github.com/openxla/xla/blob/main/xla/backends/gpu/tests/BUILD):
- load_autotune_results_using_execpath_test
- load_autotune_results_from_test_workspace_test
- dump_autotune_results_to_test_outputs_test
### Cache obsolescence
If many changes are made to a model, it is possible that the cache will no
longer contain all fusions, so the test will become slower. In this case we
would have to regenerate the autotuning cache.
If we start using a new type of GPU for running the tests, the same applies.
The cache may also become obsolete if the XLA compiler evolves and generates
different fusions.
third_party/xla/docs/pjrt/_toc.yaml
Vendored
+27
View File
@@ -0,0 +1,27 @@
# Copyright 2026 The OpenXLA Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================
toc:
- heading: PJRT
- title: Getting started
section:
- title: Introduction
path: /xla/pjrt
- title: PJRT C++ API Overview
path: /xla/pjrt/cpp_api_overview
- title: Develop a New PJRT Plugin
path: /xla/pjrt/pjrt_integration
- title: PJRT Examples
path: /xla/pjrt/examples

Some files were not shown because too many files have changed in this diff Show More