chore: import upstream snapshot with attribution

docs/get_started/overview.rst

@@ -0,0 +1,66 @@
+..  Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+..    http://www.apache.org/licenses/LICENSE-2.0
+
+..  Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+Overview
+========
+
+Apache TVM is a machine learning compilation framework, following the principle of **Python-first development**
+and **universal deployment**. It takes in pre-trained machine learning models,
+compiles and generates deployable modules that can be embedded and run everywhere. Apache TVM also enables customizing optimization processes to introduce new optimizations, libraries, codegen
+and more.
+
+Key Principle
+-------------
+
+- **Python-first**: the optimization process is fully customizable in Python.
+  It is easy to customize the optimization pipeline without recompiling the TVM stack.
+- **Composable**: the optimization process is composable. It is easy to compose
+  new optimization passes, libraries and codegen to the existing pipeline.
+
+Key Goals
+---------
+
+- **Optimize** performance of ML workloads, composing libraries and codegen.
+- **Deploy** ML workloads to a diverse set of new environments, including new runtime and new hardware.
+- **Continuously improve and customize** ML deployment pipeline in Python by quickly customizing library dispatching,
+  bringing in customized operators and code generation.
+
+Key Flow
+--------
+
+Here is a typical flow of using TVM to deploy a machine learning model. For a runnable example,
+please refer to :ref:`quick_start`
+
+1. **Import/construct an ML model**
+
+    TVM supports importing models from various frameworks, such as PyTorch and ONNX for generic ML models. Meanwhile, we can create models directly using Relax frontend for scenarios of large language models.
+
+2. **Perform composable optimization** transformations via ``pipelines``
+
+    The pipeline encapsulates a collection of transformations to achieve two goals:
+
+    - **Graph Optimizations**: such as operator fusion, and layout rewrites.
+    - **Tensor Program Optimization**: Map the operators to low-level implementations (both library or codegen)
+
+    .. note::
+
+        The two are goals but not the stages of the pipeline. The two optimizations are performed
+        **at the same level**, or separately in two stages.
+
+3. **Build and universal deploy**
+
+    Apache TVM aims to provide a universal deployment solution to bring machine learning everywhere with every language with minimum runtime support. TVM runtime can work in non-Python environments, so it works on mobile, edge devices or even bare metal devices. Additionally, TVM runtime comes with native data structures, and can also have zero copy exchange with the existing ecosystem (PyTorch, TensorFlow, TensorRT, etc.) using DLPack support.

docs/get_started/tutorials/README.txt

@@ -0,0 +1,2 @@
+Get Started
+-----------

docs/get_started/tutorials/ir_module.py

@@ -0,0 +1,285 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# ruff: noqa: E402
+
+"""
+.. _ir_module:
+
+IRModule
+========
+This tutorial presents the core abstraction of Apache TVM, the IRModule.
+The IRModule encompasses the **entirety** of the ML models, incorporating the
+computational graph, tensor programs, and potential calls to external libraries.
+
+.. contents:: Table of Contents
+    :local:
+    :depth: 1
+"""
+
+import numpy as np
+
+######################################################################
+# Create IRModule
+# ---------------
+# IRModules can be initialized in various ways. We demonstrate a few of them
+# below.
+import torch
+from torch import nn
+from torch.export import export
+
+import tvm
+from tvm import relax
+from tvm.relax.frontend.torch import from_exported_program
+
+######################################################################
+# Import from existing models
+# ~~~~~~~~~~~~~~~~~~~~~~~~~~~
+# The most common way to initialize an IRModule is to import from an existing
+# model. Apache TVM accommodates imports from a range of frameworks,
+# such as PyTorch and ONNX. This tutorial solely demonstrates the import process
+# from PyTorch.
+
+
+# Create a dummy model
+class TorchModel(nn.Module):
+    def __init__(self):
+        super().__init__()
+        self.fc1 = nn.Linear(784, 256)
+        self.relu1 = nn.ReLU()
+        self.fc2 = nn.Linear(256, 10)
+
+    def forward(self, x):
+        x = self.fc1(x)
+        x = self.relu1(x)
+        x = self.fc2(x)
+        return x
+
+
+# Give an example argument to torch.export
+example_args = (torch.randn(1, 784, dtype=torch.float32),)
+
+# Convert the model to IRModule
+with torch.no_grad():
+    exported_program = export(TorchModel().eval(), example_args)
+    mod_from_torch = from_exported_program(
+        exported_program, keep_params_as_input=True, unwrap_unit_return_tuple=True
+    )
+
+mod_from_torch, params_from_torch = relax.frontend.detach_params(mod_from_torch)
+# Print the IRModule
+mod_from_torch.show()
+
+######################################################################
+# Write with Relax NN Module
+# ~~~~~~~~~~~~~~~~~~~~~~~~~~
+# Apache TVM also provides a set of PyTorch-liked APIs, to help users
+# write the IRModule directly.
+
+from tvm.relax.frontend import nn
+
+
+class RelaxModel(nn.Module):
+    def __init__(self):
+        super().__init__()
+        self.fc1 = nn.Linear(784, 256)
+        self.relu1 = nn.ReLU()
+        self.fc2 = nn.Linear(256, 10)
+
+    def forward(self, x):
+        x = self.fc1(x)
+        x = self.relu1(x)
+        x = self.fc2(x)
+        return x
+
+
+mod_from_relax, params_from_relax = RelaxModel().export_tvm(
+    {"forward": {"x": nn.spec.Tensor((1, 784), "float32")}}
+)
+mod_from_relax.show()
+
+######################################################################
+# Create via TVMScript
+# ~~~~~~~~~~~~~~~~~~~~
+# TVMScript is a Python-based DSL for IRModules. We are able to
+# directly output the IRModule in the TVMScript syntax, or alternatively,
+# parse the TVMScript to obtain an IRModule.
+
+from tvm.script import ir as I
+from tvm.script import relax as R
+
+
+@I.ir_module
+class TVMScriptModule:
+    @R.function
+    def main(
+        x: R.Tensor((1, 784), dtype="float32"),
+        fc1_weight: R.Tensor((256, 784), dtype="float32"),
+        fc1_bias: R.Tensor((256,), dtype="float32"),
+        fc2_weight: R.Tensor((10, 256), dtype="float32"),
+        fc2_bias: R.Tensor((10,), dtype="float32"),
+    ) -> R.Tensor((1, 10), dtype="float32"):
+        R.func_attr({"num_input": 1})
+        with R.dataflow():
+            permute_dims = R.permute_dims(fc1_weight, axes=None)
+            matmul = R.matmul(x, permute_dims, out_dtype=None)
+            add = R.add(matmul, fc1_bias)
+            relu = R.nn.relu(add)
+            permute_dims1 = R.permute_dims(fc2_weight, axes=None)
+            matmul1 = R.matmul(relu, permute_dims1, out_dtype=None)
+            add1 = R.add(matmul1, fc2_bias)
+            gv = add1
+            R.output(gv)
+        return gv
+
+
+mod_from_script = TVMScriptModule
+mod_from_script.show()
+
+######################################################################
+# Attributes of an IRModule
+# -------------------------
+# An IRModule is a collection of functions, indexed by GlobalVars.
+
+mod = mod_from_torch
+print(mod.get_global_vars())
+
+######################################################################
+# We can access the functions in the IRModule by indexing with the GlobalVars
+# or their names
+
+# index by global var name
+print(mod["main"])
+# index by global var, and checking they are the same function
+(gv,) = mod.get_global_vars()
+assert mod[gv] == mod["main"]
+
+######################################################################
+# Transformations on IRModules
+# ----------------------------
+# Transformations are the import component of Apache TVM. One transformation
+# takes in an IRModule and outputs another IRModule. We can apply a sequence of
+# transformations to an IRModule to obtain a new IRModule. That is the common way to
+# optimize a model.
+#
+# In this getting started tutorial, we only demonstrate how to apply transformations
+# to an IRModule. For details of each transformation, please refer to the
+# :ref:`Transformation API Reference <api-relax-transformation>`
+
+######################################################################
+# We first apply **LegalizeOps** transformation to the IRModule. This transformation
+# will convert the Relax module into a mixed stage, with both Relax and TensorIR function
+# within the same module. Meanwhile, the Relax operators will be converted into ``call_tir``.
+
+mod = mod_from_torch
+mod = relax.transform.LegalizeOps()(mod)
+mod.show()
+
+######################################################################
+# After the transformation, there are much more functions inside the module. Let's print
+# the global vars again.
+
+print(mod.get_global_vars())
+
+######################################################################
+# Next, Apache TVM provides a set of default transformation pipelines for users,
+# to simplify the transformation process. We can then apply the default pipeline to the module.
+# The default **zero** pipeline contains very fundamental transformations, including:
+#
+# - **LegalizeOps**: This transform converts the Relax operators into `call_tir` functions
+#   with the corresponding TensorIR Functions. After this transform, the IRModule will
+#   contain both Relax functions and TensorIR functions.
+# - **AnnotateTIROpPattern**: This transform annotates the pattern of the TensorIR functions,
+#   preparing them for subsequent operator fusion.
+# - **FoldConstant**: This pass performs constant folding, optimizing operations
+#   involving constants.
+# - **FuseOps and FuseTIR**: These two passes work together to fuse operators based on the
+#   patterns annotated in the previous step (AnnotateTIROpPattern). These passes transform
+#   both Relax functions and TensorIR functions.
+#
+# .. note::
+#
+#   Here, we have applied **LegalizeOps** twice in the flow. The second time is useless but
+#   harmless.
+#
+#   Every passes can be duplicated in the flow, since we ensure the passes can handle all legal
+#   IRModule inputs. This design can help users to construct their own pipeline.
+
+mod = relax.get_pipeline("zero")(mod)
+mod.show()
+
+######################################################################
+# Deploy the IRModule Universally
+# -------------------------------
+# After the optimization, we can compile the model into a TVM runtime module.
+# Notably, Apache TVM provides the ability of universal deployment, which means
+# we can deploy the same IRModule on different backends, including CPU, GPU, and other emerging
+# backends.
+#
+# Deploy on CPU
+# ~~~~~~~~~~~~~
+# We can deploy the IRModule on CPU by specifying the target as ``llvm``.
+
+exec = tvm.compile(mod, target="llvm")
+dev = tvm.cpu()
+vm = relax.VirtualMachine(exec, dev)
+
+raw_data = np.random.rand(1, 784).astype("float32")
+data = tvm.runtime.tensor(raw_data, dev)
+cpu_out = vm["main"](data, *params_from_torch["main"]).numpy()
+print(cpu_out)
+
+######################################################################
+# Deploy on GPU
+# ~~~~~~~~~~~~~
+# Besides, CPU backend, we can also deploy the IRModule on GPU. GPU requires
+# programs containing extra information, such as the thread bindings and shared memory
+# allocations. We need a further transformation to generate the GPU programs.
+#
+# We use ``DLight`` to generate the GPU programs. In this tutorial, we won't go into
+# the details of ``DLight``.
+#
+
+from tvm.s_tir import dlight as dl
+
+with tvm.target.Target("cuda"):
+    gpu_mod = dl.ApplyDefaultSchedule(
+        dl.gpu.Matmul(),
+        dl.gpu.Fallback(),
+    )(mod)
+
+######################################################################
+# Now we can compile the IRModule on GPU, the similar way as we did on CPU.
+
+exec = tvm.compile(gpu_mod, target="cuda")
+dev = tvm.device("cuda", 0)
+vm = relax.VirtualMachine(exec, dev)
+# Need to allocate data and params on GPU device
+data = tvm.runtime.tensor(raw_data, dev)
+gpu_params = [tvm.runtime.tensor(p, dev) for p in params_from_torch["main"]]
+gpu_out = vm["main"](data, *gpu_params).numpy()
+print(gpu_out)
+
+# Check the correctness of the results
+assert np.allclose(cpu_out, gpu_out, atol=1e-3)
+
+######################################################################
+# Deploy on Other Backends
+# ~~~~~~~~~~~~~~~~~~~~~~~~
+# Apache TVM also supports other backends, such as different kinds of GPUs
+# (Metal, ROCm, Vulkan and OpenCL), different kinds of CPUs (x86, ARM), and other
+# emerging backends (e.g., WebAssembly). The deployment process is similar to the
+# GPU backend.

docs/get_started/tutorials/quick_start.py

@@ -0,0 +1,194 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# ruff: noqa: E402, E501
+
+"""
+.. _quick_start:
+
+Quick Start
+===========
+
+This tutorial is for people who are new to Apache TVM. Taking an simple example
+to show how to use Apache TVM to compile a simple neural network.
+
+.. contents:: Table of Contents
+    :local:
+    :depth: 2
+
+"""
+
+################################################################################
+# Overview
+# --------
+# Apache TVM is a machine learning compilation framework, following the principle of
+# **Python-first development** and **universal deployment**. It takes in pre-trained
+# machine learning models, compiles and generates deployable modules that can be embedded
+# and run everywhere.
+# Apache TVM also enables customizing optimization processes to introduce new optimizations,
+# libraries, codegen and more.
+#
+# Apache TVM can help to:
+#
+# - **Optimize** performance of ML workloads, composing libraries and codegen.
+# - **Deploy** ML workloads to a diverse set of new environments, including new runtime and new
+#   hardware.
+# - **Continuously improve and customize** ML deployment pipeline in Python by quickly customizing
+#   library dispatching, bringing in customized operators and code generation.
+
+################################################################################
+# Overall Flow
+# ------------
+# Then we will show the overall flow of using Apache TVM to compile a neural network model,
+# showing how to optimize, deploy and run the model.
+# The overall flow is illustrated as the figure:
+#
+# .. figure:: https://raw.githubusercontent.com/tlc-pack/web-data/main/images/design/tvm_overall_flow.svg
+#    :align: center
+#    :width: 80%
+#
+# The overall flow consists of the following steps:
+#
+# - **Construct or Import a Model**: Construct a neural network model or import a pre-trained
+#   model from other frameworks (e.g. PyTorch, ONNX), and create the TVM IRModule, which contains
+#   all the information needed for compilation, including high-level Relax functions for
+#   computational graph, and low-level TensorIR functions for tensor program.
+# - **Perform Composable Optimizations**: Perform a series of optimization transformations,
+#   such as graph optimizations, tensor program optimizations, and library dispatching.
+# - **Build and Universal Deployment**: Build the optimized model to a deployable module to the
+#   universal runtime, and execute it on different devices, such as CPU, GPU, or other accelerators.
+
+################################################################################
+# Construct or Import a Model
+# ~~~~~~~~~~~~~~~~~~~~~~~~~~~
+# Before we get started, let's construct a neural network model first.
+# In this tutorial, to make things simple, we will define a two-layer MLP network
+# directly in this script with the TVM Relax frontend, which is a similar API to PyTorch.
+#
+
+import tvm
+from tvm import relax
+from tvm.relax.frontend import nn
+
+
+class MLPModel(nn.Module):
+    def __init__(self):
+        super().__init__()
+        self.fc1 = nn.Linear(784, 256)
+        self.relu1 = nn.ReLU()
+        self.fc2 = nn.Linear(256, 10)
+
+    def forward(self, x):
+        x = self.fc1(x)
+        x = self.relu1(x)
+        x = self.fc2(x)
+        return x
+
+
+################################################################################
+# Then we can export the model to TVM IRModule, which is the central intermediate representation
+# in TVM.
+
+mod, param_spec = MLPModel().export_tvm(
+    spec={"forward": {"x": nn.spec.Tensor((1, 784), "float32")}}
+)
+mod.show()
+
+################################################################################
+# Perform Optimization Transformations
+# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+# Apache TVM leverage ``pipeline`` to transform and optimize program.
+# The pipeline encapsulates a collection of transformation that gets two goals (at the same level):
+#
+# - **Model optimizations**: such as operator fusion, layout rewrites.
+# - **Tensor program optimization**: Map the operators to low-level implementations
+#   (both library or codegen)
+#
+# .. note::
+#   The twos are goals but not the stages of the pipeline. The two optimizations are performed
+#   **at the same level**, or separately in two stages.
+#
+# .. note::
+#   In this tutorial we only demonstrate the overall flow, by leverage ``zero`` optimization
+#   pipeline, instead of optimizing for any specific target.
+
+mod = relax.get_pipeline("zero")(mod)
+
+
+################################################################################
+# Build and Universal Deployment
+# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+# After the optimization, we can build the model to a deployable module and run it on
+# different devices.
+
+
+import numpy as np
+
+target = tvm.target.Target("llvm")
+ex = tvm.compile(mod, target)
+device = tvm.cpu()
+vm = relax.VirtualMachine(ex, device)
+data = np.random.rand(1, 784).astype("float32")
+tvm_data = tvm.runtime.tensor(data, device=device)
+params = [np.random.rand(*param.shape).astype("float32") for _, param in param_spec]
+params = [tvm.runtime.tensor(param, device=device) for param in params]
+print(vm["forward"](tvm_data, *params).numpy())
+
+################################################################################
+# Our goal is to bring machine learning to the application with any language of interest,
+# with the minimum runtime support.
+#
+# - Each function in IRModule becomes a runnable function in the runtime. For example in LLM
+#   cases, we can call ``prefill`` and ``decode`` functions directly.
+#
+#   .. code-block:: Python
+#
+#       prefill_logits = vm["prefill"](inputs, weight, kv_cache)
+#       decoded_logits = vm["decode"](inputs, weight, kv_cache)
+#
+# - TVM runtime comes with native data structures, such as Tensor, can also have zero
+#   copy exchange with existing ecosystem (DLPack exchange with PyTorch)
+#
+#   .. code-block:: Python
+#
+#       # Convert PyTorch tensor to TVM Tensor
+#       x_tvm = tvm.runtime.from_dlpack(x_torch)
+#       # Convert TVM Tensor to PyTorch tensor
+#       x_torch = torch.from_dlpack(x_tvm)
+#
+# - TVM runtime works in non-python environments, so it works on settings such as mobile
+#
+#   .. code-block:: C++
+#
+#       // C++ snippet
+#       runtime::Module vm = ex.GetFunction("load_executable")();
+#       vm.GetFunction("init")(...);
+#       Tensor out = vm.GetFunction("prefill")(data, weight, kv_cache);
+#
+#   .. code-block:: Java
+#
+#       // Java snippet
+#       Module vm = ex.getFunction("load_executable").invoke();
+#       vm.getFunction("init").pushArg(...).invoke;
+#       Tensor out = vm.getFunction("prefill").pushArg(data).pushArg(weight).pushArg(kv_cache).invoke();
+#
+
+################################################################################
+# Read next
+# ---------
+# This tutorial demonstrates the overall flow of using Apache TVM to compile a neural network model.
+# For more advanced or specific topics, please refer to the following tutorials
+#