wehub/tensorflow--tensorflow
1
0
Fork 0
Code Issues Pull Requests Actions 5 Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution
cffconvert / validate (push) Has been skipped
Details
License Check / license-check (push) Failing after 2s
Details

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 12:14:16 +08:00
commit 8a852e4b4e
36502 changed files with 9277225 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tensorflow/lite/g3doc/guide/authoring.ipynb
+343
View File
@@ -0,0 +1,343 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "g_nWetWWd_ns"
},
"source": [
"##### Copyright 2024 The AI Edge Authors."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "2pHVBk_seED1"
},
"outputs": [],
"source": [
"#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "M7vSdG6sAIQn"
},
"source": [
"# TFLite Authoring Tool"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "fwc5GKHBASdc"
},
"source": [
"<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
" <td>\n",
" <a target=\"_blank\" href=\"https://www.tensorflow.org/lite/guide/authoring\"><img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" />View on TensorFlow.org</a>\n",
" </td>\n",
" <td>\n",
" <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/guide/authoring.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
" </td>\n",
" <td>\n",
" <a target=\"_blank\" href=\"https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/guide/authoring.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://storage.googleapis.com/tensorflow_docs/tensorflow/tensorflow/lite/g3doc/guide/authoring.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\" />Download notebook</a>\n",
" </td>\n",
"</table>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "9ee074e4"
},
"source": [
"TensorFlow Lite Authoring API provides a way to maintain your `tf.function` models compatibile with TensorFlow Lite.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "UaWdLA3fQDK2"
},
"source": [
"## Setup"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "DWjLcy2CvgxH"
},
"outputs": [],
"source": [
"import tensorflow as tf"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "CmkXJRDj5hTi"
},
"source": [
"## TensorFlow to TensorFlow Lite compatibility issue\n",
"\n",
"If you want to use your TF model on devices, you need to convert it to a TFLite model to use it from TFLite interpreter.\n",
"During the conversion, you might encounter a compatibility error because of unsupported TensorFlow ops by the TFLite builtin op set.\n",
"\n",
"This is a kind of annoying issue. How can you detect it earlier like the model authoring time?\n",
"\n",
"Note that the following code will fail on the `converter.convert()` call.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "LHKqKFm5OvyQ"
},
"outputs": [],
"source": [
"@tf.function(input_signature=[\n",
" tf.TensorSpec(shape=[None], dtype=tf.float32)\n",
"])\n",
"def f(x):\n",
" return tf.cosh(x)\n",
"\n",
"# Evaluate the tf.function\n",
"result = f(tf.constant([0.0]))\n",
"print (f\"result = {result}\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "BS5bOoD50zaU"
},
"outputs": [],
"source": [
"# Convert the tf.function\n",
"converter = tf.lite.TFLiteConverter.from_concrete_functions(\n",
" [f.get_concrete_function()], f)\n",
"try:\n",
" fb_model = converter.convert()\n",
"except Exception as e:\n",
" print(f\"Got an exception: {e}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "eLU0Y9V8g_Wk"
},
"source": [
"## Simple Target Aware Authoring usage\n",
"\n",
"We introduced Authoring API to detect the TensorFlow Lite compatibility issue during the model authoring time.\n",
"\n",
"You just need to add `@tf.lite.experimental.authoring.compatible` decorator to wrap your `tf.function` model to check TFLite compatibility.\n",
"\n",
"After this, the compatibility will be checked automatically when you evaluate your model."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "zVSh6VCDhbPz"
},
"outputs": [],
"source": [
"@tf.lite.experimental.authoring.compatible\n",
"@tf.function(input_signature=[\n",
" tf.TensorSpec(shape=[None], dtype=tf.float32)\n",
"])\n",
"def f(x):\n",
" return tf.cosh(x)\n",
"\n",
"# Evaluate the tf.function\n",
"result = f(tf.constant([0.0]))\n",
"print (f\"result = {result}\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ZWkBEqv-eUwV"
},
"source": [
"If any TensorFlow Lite compatibility issue is found, it will show `COMPATIBILITY WARNING` or `COMPATIBILITY ERROR` with the exact location of the problematic op. In this example, it shows the location of `tf.Cosh` op in your tf.function model.\n",
"\n",
"You can also check the compatiblity log with the `<function_name>.get_compatibility_log()` method."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "irwO2qdv2RPA"
},
"outputs": [],
"source": [
"compatibility_log = '\\n'.join(f.get_compatibility_log())\n",
"print (f\"compatibility_log = {compatibility_log}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "-LTVE00CiqpS"
},
"source": [
"## Raise an exception for an incompatibility\n",
"\n",
"You can provide an option to the `@tf.lite.experimental.authoring.compatible` decorator. The `raise_exception` option gives you an exception when you're trying to evaluate the decorated model."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "YfPfOJm5jST4"
},
"outputs": [],
"source": [
"@tf.lite.experimental.authoring.compatible(raise_exception=True)\n",
"@tf.function(input_signature=[\n",
" tf.TensorSpec(shape=[None], dtype=tf.float32)\n",
"])\n",
"def f(x):\n",
" return tf.cosh(x)\n",
"\n",
"# Evaluate the tf.function\n",
"try:\n",
" result = f(tf.constant([0.0]))\n",
" print (f\"result = {result}\")\n",
"except Exception as e:\n",
" print(f\"Got an exception: {e}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "WXywHrR0Xjop"
},
"source": [
"## Specifying \"Select TF ops\" usage\n",
"\n",
"If you're already aware of [Select TF ops](https://www.tensorflow.org/lite/guide/ops_select) usage, you can tell this to the Authoring API by setting `converter_target_spec`. It's the same [tf.lite.TargetSpec](https://www.tensorflow.org/api_docs/python/tf/lite/TargetSpec) object you'll use it for [tf.lite.TFLiteConverter](https://www.tensorflow.org/api_docs/python/tf/lite/TFLiteConverter) API.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "B483OwYQYG8A"
},
"outputs": [],
"source": [
"target_spec = tf.lite.TargetSpec()\n",
"target_spec.supported_ops = [\n",
" tf.lite.OpsSet.TFLITE_BUILTINS,\n",
" tf.lite.OpsSet.SELECT_TF_OPS,\n",
"]\n",
"@tf.lite.experimental.authoring.compatible(converter_target_spec=target_spec, raise_exception=True)\n",
"@tf.function(input_signature=[\n",
" tf.TensorSpec(shape=[None], dtype=tf.float32)\n",
"])\n",
"def f(x):\n",
" return tf.cosh(x)\n",
"\n",
"# Evaluate the tf.function\n",
"result = f(tf.constant([0.0]))\n",
"print (f\"result = {result}\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "mtept13-C6uD"
},
"source": [
"## Checking GPU compatibility\n",
"\n",
"If you want to ensure your model is compatibile with [GPU delegate](https://www.tensorflow.org/lite/performance/gpu) of TensorFlow Lite, you can set `experimental_supported_backends` of [tf.lite.TargetSpec](https://www.tensorflow.org/api_docs/python/tf/lite/TargetSpec).\n",
"\n",
"The following example shows how to ensure GPU delegate compatibility of your model. Note that this model has compatibility issues since it uses a 2D tensor with tf.slice operator and unsupported tf.cosh operator. You'll see two `COMPATIBILITY WARNING` with the location information."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "_DzHV3KVC0T0"
},
"outputs": [],
"source": [
"target_spec = tf.lite.TargetSpec()\n",
"target_spec.supported_ops = [\n",
" tf.lite.OpsSet.TFLITE_BUILTINS,\n",
" tf.lite.OpsSet.SELECT_TF_OPS,\n",
"]\n",
"target_spec.experimental_supported_backends = [\"GPU\"]\n",
"@tf.lite.experimental.authoring.compatible(converter_target_spec=target_spec)\n",
"@tf.function(input_signature=[\n",
" tf.TensorSpec(shape=[4, 4], dtype=tf.float32)\n",
"])\n",
"def func(x):\n",
" y = tf.cosh(x)\n",
" return y + tf.slice(x, [1, 1], [1, 1])\n",
"\n",
"result = func(tf.ones(shape=(4,4), dtype=tf.float32))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JvLEtCWRvvy8"
},
"source": [
"## Read more\n",
"\n",
"For more information, please refer to:\n",
"- [tf.function](https://www.tensorflow.org/api_docs/python/tf/function) API doc\n",
"- [Better performance with tf.function](https://www.tensorflow.org/guide/function)\n",
"- [TensorFlow Lite converter](https://www.tensorflow.org/lite/models/convert)\n",
"- [TensorFlow Lite Model Analyzer](https://www.tensorflow.org/lite/guide/model_analyzer)\n",
"- [TensorFlow Lite GPU delegate](https://www.tensorflow.org/lite/performance/gpu)"
]
}
],
"metadata": {
"colab": {
"collapsed_sections": [],
"name": "authoring.ipynb",
"provenance": [],
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
tensorflow/lite/g3doc/guide/build_arm.md
+105
View File
@@ -0,0 +1,105 @@
# Build TensorFlow Lite for ARM boards
This page describes how to build the TensorFlow Lite libraries for ARM-based
computers.
TensorFlow Lite supports two build systems and supported features from each
build system are not identical. Check the following table to pick a proper build
system.
Feature | Bazel | CMake
----------------------------------------------------------------------------------------- | ---------------------------- | -----
Predefined toolchains | armhf, aarch64 | armel, armhf, aarch64
Custom toolchains | harder to use | easy to use
[Select TF ops](https://www.tensorflow.org/lite/guide/ops_select) | supported | not supported
[GPU delegate](https://www.tensorflow.org/lite/performance/gpu) | only available for Android | any platform that supports OpenCL
XNNPack | supported | supported
[Python Wheel](https://www.tensorflow.org/lite/guide/build_cmake_pip) | supported | supported
[C API](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/c/README.md) | supported | [supported](https://www.tensorflow.org/lite/guide/build_cmake#build_tensorflow_lite_c_library)
[C++ API](https://www.tensorflow.org/lite/guide/inference#load_and_run_a_model_in_c) | supported for Bazel projects | supported for CMake projects
## Cross-compilation for ARM with CMake
If you have a CMake project or if you want to use a custom toolchain, you'd
better use CMake for cross compilation. There is a separate
[Cross compilation TensorFlow Lite with CMake](https://www.tensorflow.org/lite/guide/build_cmake_arm)
page available for this.
## Cross-compilation for ARM with Bazel
If you have a Bazel project or if you want to use TF ops, you'd better use Bazel
build system. You'll use the integrated
[ARM GCC 8.3 toolchains](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/tools/toolchains/embedded/arm-linux)
with Bazel to build an ARM32/64 shared library.
| Target Architecture | Bazel Configuration | Compatible Devices |
| ------------------- | ----------------------- | -------------------------- |
| armhf (ARM32) | --config=elinux_armhf | RPI3, RPI4 with 32 bit |
: : : Raspberry Pi OS :
| AArch64 (ARM64) | --config=elinux_aarch64 | Coral, RPI4 with Ubuntu 64 |
: : : bit :
Note: The generated shared library requires glibc 2.28 or higher to run.
The following instructions have been tested on Ubuntu 16.04.3 64-bit PC (AMD64)
and TensorFlow devel docker image
[tensorflow/tensorflow:devel](https://hub.docker.com/r/tensorflow/tensorflow/tags/).
To cross compile TensorFlow Lite with Bazel, follow the steps:
#### Step 1. Install Bazel
Bazel is the primary build system for TensorFlow. Install the latest version of
the [Bazel build system](https://bazel.build/versions/master/docs/install.html).
**Note:** If you're using the TensorFlow Docker image, Bazel is already
available.
#### Step 2. Clone TensorFlow repository
```sh
git clone https://github.com/tensorflow/tensorflow.git tensorflow_src
```
**Note:** If you're using the TensorFlow Docker image, the repo is already
provided in `/tensorflow_src/`.
#### Step 3. Build ARM binary
##### C library
```bash
bazel build --config=elinux_aarch64 -c opt //tensorflow/lite/c:libtensorflowlite_c.so
```
You can find a shared library in:
`bazel-bin/tensorflow/lite/c/libtensorflowlite_c.so`.
**Note:** Use `elinux_armhf` for
[32bit ARM hard float](https://wiki.debian.org/ArmHardFloatPort) build.
Check
[TensorFlow Lite C API](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/c/README.md)
page for the detail.
##### C++ library
```bash
bazel build --config=elinux_aarch64 -c opt //tensorflow/lite:libtensorflowlite.so
```
You can find a shared library in:
`bazel-bin/tensorflow/lite/libtensorflowlite.so`.
Currently, there is no straightforward way to extract all header files needed,
so you must include all header files in tensorflow/lite/ from the TensorFlow
repository. Additionally, you will need header files from FlatBuffers and
Abseil.
##### Etc
You can also build other Bazel targets with the toolchain. Here are some useful
targets.
* //tensorflow/lite/tools/benchmark:benchmark_model
* //tensorflow/lite/examples/label_image:label_image
tensorflow/lite/g3doc/guide/build_cmake.md
+315
View File
@@ -0,0 +1,315 @@
# Build TensorFlow Lite with CMake
This page describes how to build and use the TensorFlow Lite library with
[CMake](https://cmake.org/) tool.
The following instructions have been tested on Ubuntu 16.04.3 64-bit PC (AMD64)
, macOS Catalina (x86_64), Windows 10 and TensorFlow devel Docker image
[tensorflow/tensorflow:devel](https://hub.docker.com/r/tensorflow/tensorflow/tags/).
**Note:** This feature is available since version 2.4.
### Step 1. Install CMake tool
It requires CMake 3.16 or higher. On Ubuntu, you can simply run the following
command.
```sh
sudo apt-get install cmake
```
Or you can follow
[the official cmake installation guide](https://cmake.org/install/)
### Step 2. Clone TensorFlow repository
```sh
git clone https://github.com/tensorflow/tensorflow.git tensorflow_src
```
**Note:** If you're using the TensorFlow Docker image, the repo is already
provided in `/tensorflow_src/`.
### Step 3. Create CMake build directory
```sh
mkdir tflite_build
cd tflite_build
```
### Step 4. Run CMake tool with configurations
#### Release build
It generates an optimized release binary by default. If you want to build for
your workstation, simply run the following command.
```sh
cmake ../tensorflow_src/tensorflow/lite
```
#### Debug build
If you need to produce a debug build which has symbol information, you need to
provide the `-DCMAKE_BUILD_TYPE=Debug` option.
```sh
cmake ../tensorflow_src/tensorflow/lite -DCMAKE_BUILD_TYPE=Debug
```
#### Build with kernel unit tests
In order to be able to run kernel tests, you need to provide the
`-DTFLITE_KERNEL_TEST=on` flag. Unit test cross-compilation specifics can be
found in the next subsection.
```sh
cmake ../tensorflow_src/tensorflow/lite -DTFLITE_KERNEL_TEST=on
```
#### Build installable package
To build an installable package that can be used as a dependency by another
CMake project with `find_package(tensorflow-lite CONFIG)`, use the
`-DTFLITE_ENABLE_INSTALL=ON` option.
You should ideally also provide your own versions of library dependencies.
These will also need to used by the project that depends on TF Lite. You can
use the `-DCMAKE_FIND_PACKAGE_PREFER_CONFIG=ON` and set the `<PackageName>_DIR`
variables to point to your library installations.
```sh
cmake ../tensorflow_src/tensorflow/lite -DTFLITE_ENABLE_INSTALL=ON \
-DCMAKE_FIND_PACKAGE_PREFER_CONFIG=ON \
-DSYSTEM_FARMHASH=ON \
-DSYSTEM_PTHREADPOOL=ON \
-Dabsl_DIR=<install path>/lib/cmake/absl \
-DEigen3_DIR=<install path>/share/eigen3/cmake \
-DFlatBuffers_DIR=<install path>/lib/cmake/flatbuffers \
-Dgemmlowp_DIR=<install path>/lib/cmake/gemmlowp \
-DNEON_2_SSE_DIR=<install path>/lib/cmake/NEON_2_SSE \
-Dcpuinfo_DIR=<install path>/share/cpuinfo \
-Druy_DIR=<install path>/lib/cmake/ruy
```
**Note:** Refer to CMake documentation for
[`find_package`](https://cmake.org/cmake/help/latest/command/find_package.html)
to learn more about handling and locating packages.
#### Cross-compilation
You can use CMake to build binaries for ARM64 or Android target architectures.
In order to cross-compile the TF Lite, you namely need to provide the path to
the SDK (e.g. ARM64 SDK or NDK in Android's case) with `-DCMAKE_TOOLCHAIN_FILE`
flag.
```sh
cmake -DCMAKE_TOOLCHAIN_FILE=<CMakeToolchainFileLoc> ../tensorflow/lite/
```
##### Specifics of Android cross-compilation
For Android cross-compilation, you need to install
[Android NDK](https://developer.android.com/ndk) and provide the NDK path with
`-DCMAKE_TOOLCHAIN_FILE` flag mentioned above. You also need to set target ABI
with`-DANDROID_ABI` flag.
```sh
cmake -DCMAKE_TOOLCHAIN_FILE=<NDK path>/build/cmake/android.toolchain.cmake \
-DANDROID_ABI=arm64-v8a ../tensorflow_src/tensorflow/lite
```
##### Specifics of kernel (unit) tests cross-compilation
Cross-compilation of the unit tests requires flatc compiler for the host
architecture. For this purpose, there is a CMakeLists located in
`tensorflow/lite/tools/cmake/native_tools/flatbuffers` to build the flatc
compiler with CMake in advance in a separate build directory using the host
toolchain.
```sh
mkdir flatc-native-build && cd flatc-native-build
cmake ../tensorflow_src/tensorflow/lite/tools/cmake/native_tools/flatbuffers
cmake --build .
```
It is also possible **to install** the *flatc* to a custom installation location
(e.g. to a directory containing other natively-built tools instead of the CMake
build directory):
```sh
cmake -DCMAKE_INSTALL_PREFIX=<native_tools_dir> ../tensorflow_src/tensorflow/lite/tools/cmake/native_tools/flatbuffers
cmake --build .
```
For the TF Lite cross-compilation itself, additional parameter
`-DTFLITE_HOST_TOOLS_DIR=<flatc_dir_path>` pointing to the directory containing
the native *flatc* binary needs to be provided along with the
`-DTFLITE_KERNEL_TEST=on` flag mentioned above.
```sh
cmake -DCMAKE_TOOLCHAIN_FILE=${OE_CMAKE_TOOLCHAIN_FILE} -DTFLITE_KERNEL_TEST=on -DTFLITE_HOST_TOOLS_DIR=<flatc_dir_path> ../tensorflow/lite/
```
##### Cross-compiled kernel (unit) tests launch on target
Unit tests can be run as separate executables or using the CTest utility. As far
as CTest is concerned, if at least one of the parameters `TFLITE_ENABLE_NNAPI,
TFLITE_ENABLE_XNNPACK` or `TFLITE_EXTERNAL_DELEGATE` is enabled for the TF Lite
build, the resulting tests are generated with two different **labels**
(utilizing the same test executable): - *plain* - denoting the tests ones run on
CPU backend - *delegate* - denoting the tests expecting additional launch
arguments used for the used delegate specification
Both `CTestTestfile.cmake` and `run-tests.cmake` (as referred below) are
available in `<build_dir>/kernels`.
Launch of unit tests with CPU backend (provided the `CTestTestfile.cmake` is
present on target in the current directory):
```sh
ctest -L plain
```
Launch examples of unit tests using delegates (provided the
`CTestTestfile.cmake` as well as `run-tests.cmake` file are present on target in
the current directory):
```sh
cmake -E env TESTS_ARGUMENTS=--use_nnapi=true\;--nnapi_accelerator_name=vsi-npu ctest -L delegate
cmake -E env TESTS_ARGUMENTS=--use_xnnpack=true ctest -L delegate
cmake -E env TESTS_ARGUMENTS=--external_delegate_path=<PATH> ctest -L delegate
```
**A known limitation** of this way of providing additional delegate-related
launch arguments to unit tests is that it effectively supports only those with
an **expected return value of 0**. Different return values will be reported as a
test failure.
#### OpenCL GPU delegate
If your target machine has OpenCL support, you can use
[GPU delegate](https://www.tensorflow.org/lite/performance/gpu) which can
leverage your GPU power.
To configure OpenCL GPU delegate support:
```sh
cmake ../tensorflow_src/tensorflow/lite -DTFLITE_ENABLE_GPU=ON
```
**Note:** It's experimental and available starting from TensorFlow 2.5. There
could be compatibility issues. It's only verified with Android devices and
NVidia CUDA OpenCL 1.2.
### Step 5. Build TensorFlow Lite
In the `tflite_build` directory,
```sh
cmake --build . -j
```
**Note:** This generates a static library `libtensorflow-lite.a` in the current
directory but the library isn't self-contained since all the transitive
dependencies are not included. To use the library properly, you need to create a
CMake project. Please refer the
["Create a CMake project which uses TensorFlow Lite"](#create_a_cmake_project_which_uses_tensorflow_lite)
section.
### Step 6. Build TensorFlow Lite Benchmark Tool and Label Image Example (Optional)
In the `tflite_build` directory,
```sh
cmake --build . -j -t benchmark_model
```
```sh
cmake --build . -j -t label_image
```
## Available Options to build TensorFlow Lite
Here is the list of available options. You can override it with
`-D<option_name>=[ON|OFF]`. For example, `-DTFLITE_ENABLE_XNNPACK=OFF` to
disable XNNPACK which is enabled by default.
| Option Name | Feature | Android | Linux | macOS | Windows |
| ----------------------- | -------------- | ------- | ----- | ----- | ------- |
| `TFLITE_ENABLE_RUY` | Enable RUY | ON | OFF | OFF | OFF |
: : matrix : : : : :
: : multiplication : : : : :
: : library : : : : :
| `TFLITE_ENABLE_NNAPI` | Enable NNAPI | ON | OFF | N/A | N/A |
: : delegate : : : : :
| `TFLITE_ENABLE_GPU` | Enable GPU | OFF | OFF | N/A | N/A |
: : delegate : : : : :
| `TFLITE_ENABLE_XNNPACK` | Enable XNNPACK | ON | ON | ON | ON |
: : delegate : : : : :
| `TFLITE_ENABLE_MMAP` | Enable MMAP | ON | ON | ON | N/A |
## Create a CMake project which uses TensorFlow Lite
Here is the CMakeLists.txt of
[TFLite minimal example](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/examples/minimal).
You need to have add_subdirectory() for TensorFlow Lite directory and link
`tensorflow-lite` with target_link_libraries().
```
cmake_minimum_required(VERSION 3.16)
project(minimal C CXX)
set(TENSORFLOW_SOURCE_DIR "" CACHE PATH
"Directory that contains the TensorFlow project" )
if(NOT TENSORFLOW_SOURCE_DIR)
get_filename_component(TENSORFLOW_SOURCE_DIR
"${CMAKE_CURRENT_LIST_DIR}/../../../../" ABSOLUTE)
endif()
add_subdirectory(
"${TENSORFLOW_SOURCE_DIR}/tensorflow/lite"
"${CMAKE_CURRENT_BINARY_DIR}/tensorflow-lite" EXCLUDE_FROM_ALL)
add_executable(minimal minimal.cc)
target_link_libraries(minimal tensorflow-lite)
```
## Build TensorFlow Lite C library
If you want to build TensorFlow Lite shared library for
[C API](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/c/README.md),
follow [step 1](#step-1-install-cmake-tool) to
[step 3](#step-3-create-cmake-build-directory) first. After that, run the
following commands.
### Linux / MacOS
```sh
cmake ../tensorflow_src/tensorflow/lite/c
cmake --build . -j
```
### Windows
```sh
cmake ../tensorflow_src/tensorflow/lite/c
cmake --build . -j --config Release
```
### Compiled Library
The above command generates the following shared library in the current
directory.
Platform | Library name
-------- | ---------------------------
Linux | `libtensorflowlite_c.so`
macOS | `libtensorflowlite_c.dylib`
Windows | `tensorflowlite_c.dll`
**Note:** You need the public headers (`tensorflow/lite/c_api.h`,
`tensorflow/lite/c_api_experimental.h`, `tensorflow/lite/c_api_types.h`, and
`tensorflow/lite/common.h`), and the private headers that those public headers
include (`tensorflow/lite/core/builtin_ops.h`, `tensorflow/lite/core/c/*.h`, and
`tensorflow/lite/core/async/c/*.h`, ) to use the generated shared library.
tensorflow/lite/g3doc/guide/build_cmake_arm.md
+179
View File
@@ -0,0 +1,179 @@
# Cross compilation TensorFlow Lite with CMake
This page describes how to build the TensorFlow Lite library for various ARM
devices.
The following instructions have been tested on Ubuntu 16.04.3 64-bit PC (AMD64)
, TensorFlow devel docker image
[tensorflow/tensorflow:devel](https://hub.docker.com/r/tensorflow/tensorflow/tags/).
**Note:** This feature is available since version 2.4.
### Prerequisites
You need CMake installed and downloaded TensorFlow source code. Please check
[Build TensorFlow Lite with CMake](https://www.tensorflow.org/lite/guide/build_cmake)
page for the details.
### Check your target environment
The following examples are tested under Raspberry Pi OS, Ubuntu Server 20.04 LTS
and Mendel Linux 4.0. Depending on your target glibc version and CPU
capabilities, you may need to use different version of toolchain and build
parameters.
#### Checking glibc version
```sh
ldd --version
```
<pre class="tfo-notebook-code-cell-output">
ldd (Debian GLIBC 2.28-10) 2.28
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.
</pre>
#### Checking ABI compatibility
If your target is ARM 32-bit, there are two ABI available depending on VFP
availity. [armhf](https://wiki.debian.org/ArmHardFloatPort) and
[armel](https://wiki.debian.org/ArmEabiPort). This document shows an armhf
example, you need to use different toolchain for armel targets.
#### Checking CPU capability
For ARMv7, you should know target's supported VFP version and NEON availability.
```sh
cat /proc/cpuinfo
```
<pre class="tfo-notebook-code-cell-output">
processor : 0
model name : ARMv7 Processor rev 3 (v7l)
BogoMIPS : 108.00
Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x0
CPU part : 0xd08
CPU revision : 3
</pre>
## Build for AArch64 (ARM64)
This instruction shows how to build AArch64 binary which is compatible with
[Coral Mendel Linux 4.0](https://coral.ai/), Raspberry Pi (with
[Ubuntu Server 20.04.01 LTS 64-bit](https://ubuntu.com/download/raspberry-pi)
installed).
#### Download toolchain
These commands install `gcc-arm-8.3-2019.03-x86_64-aarch64-linux-gnu` toolchain
under ${HOME}/toolchains.
```sh
curl -LO https://storage.googleapis.com/mirror.tensorflow.org/developer.arm.com/media/Files/downloads/gnu-a/8.3-2019.03/binrel/gcc-arm-8.3-2019.03-x86_64-aarch64-linux-gnu.tar.xz
mkdir -p ${HOME}/toolchains
tar xvf gcc-arm-8.3-2019.03-x86_64-aarch64-linux-gnu.tar.xz -C ${HOME}/toolchains
```
**Note:** Binaries built with GCC 8.3 require glibc 2.28 or higher. If your
target has lower glibc version, you need to use older GCC toolchain.
#### Run CMake
```sh
ARMCC_PREFIX=${HOME}/toolchains/gcc-arm-8.3-2019.03-x86_64-aarch64-linux-gnu/bin/aarch64-linux-gnu-
ARMCC_FLAGS="-funsafe-math-optimizations"
cmake -DCMAKE_C_COMPILER=${ARMCC_PREFIX}gcc \
-DCMAKE_CXX_COMPILER=${ARMCC_PREFIX}g++ \
-DCMAKE_C_FLAGS="${ARMCC_FLAGS}" \
-DCMAKE_CXX_FLAGS="${ARMCC_FLAGS}" \
-DCMAKE_VERBOSE_MAKEFILE:BOOL=ON \
-DCMAKE_SYSTEM_NAME=Linux \
-DCMAKE_SYSTEM_PROCESSOR=aarch64 \
../tensorflow/lite/
```
**Note:** You can enable GPU delegate with `-DTFLITE_ENABLE_GPU=ON` if your
target device supports OpenCL 1.2 or higher.
## Build for ARMv7 NEON enabled
This instruction shows how to build ARMv7 with VFPv4 and NEON enabled binary
which is compatible with Raspberry Pi 3 and 4.
#### Download toolchain
These commands install `gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf`
toolchain under ${HOME}/toolchains.
```sh
curl -LO https://storage.googleapis.com/mirror.tensorflow.org/developer.arm.com/media/Files/downloads/gnu-a/8.3-2019.03/binrel/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf.tar.xz
mkdir -p ${HOME}/toolchains
tar xvf gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf.tar.xz -C ${HOME}/toolchains
```
**Note:** Binaries built with GCC 8.3 require glibc 2.28 or higher. If your
target has lower glibc version, you need to use older GCC toolchain.
#### Run CMake
```sh
ARMCC_FLAGS="-march=armv7-a -mfpu=neon-vfpv4 -funsafe-math-optimizations -mfp16-format=ieee"
ARMCC_PREFIX=${HOME}/toolchains/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/bin/arm-linux-gnueabihf-
cmake -DCMAKE_C_COMPILER=${ARMCC_PREFIX}gcc \
-DCMAKE_CXX_COMPILER=${ARMCC_PREFIX}g++ \
-DCMAKE_C_FLAGS="${ARMCC_FLAGS}" \
-DCMAKE_CXX_FLAGS="${ARMCC_FLAGS}" \
-DCMAKE_VERBOSE_MAKEFILE:BOOL=ON \
-DCMAKE_SYSTEM_NAME=Linux \
-DCMAKE_SYSTEM_PROCESSOR=armv7 \
../tensorflow/lite/
```
**Note:** Since ARMv7 architecture is diverse, you may need to update
`ARMCC_FLAGS` for your target device profiles. For example, when compiling with
XNNPACK enabled (i.e. `XNNPACK=ON`) in Tensorflow Lite 2.8, please add
`-mfp16-format=ieee` to `ARMCC_FLAGS`.
## Build for Raspberry Pi Zero (ARMv6)
This instruction shows how to build ARMv6 binary which is compatible with
Raspberry Pi Zero.
#### Download toolchain
These commands install `gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf`
toolchain under ${HOME}/toolchains.
```sh
curl -LO https://storage.googleapis.com/mirror.tensorflow.org/developer.arm.com/media/Files/downloads/gnu-a/8.3-2019.03/binrel/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf.tar.xz
mkdir -p ${HOME}/toolchains
tar xvf gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf.tar.xz -C ${HOME}/toolchains
```
**Note:** Binaries built with GCC 8.3 require glibc 2.28 or higher. If your
target has lower glibc version, you need to use older GCC toolchain.
#### Run CMake
```sh
ARMCC_FLAGS="-march=armv6 -mfpu=vfp -mfloat-abi=hard -funsafe-math-optimizations"
ARMCC_PREFIX=${HOME}/toolchains/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/bin/arm-linux-gnueabihf-
cmake -DCMAKE_C_COMPILER=${ARMCC_PREFIX}gcc \
-DCMAKE_CXX_COMPILER=${ARMCC_PREFIX}g++ \
-DCMAKE_C_FLAGS="${ARMCC_FLAGS}" \
-DCMAKE_CXX_FLAGS="${ARMCC_FLAGS}" \
-DCMAKE_VERBOSE_MAKEFILE:BOOL=ON \
-DCMAKE_SYSTEM_NAME=Linux \
-DCMAKE_SYSTEM_PROCESSOR=armv6 \
-DTFLITE_ENABLE_XNNPACK=OFF \
../tensorflow/lite/
```
**Note:** XNNPACK is disabled since there is no NEON support.
tensorflow/lite/g3doc/guide/build_cmake_pip.md
+94
View File
@@ -0,0 +1,94 @@
# Build TensorFlow Lite Python Wheel Package
This page describes how to build the TensorFlow Lite `tflite_runtime` Python
library for x86_64 and various ARM devices.
The following instructions have been tested on Ubuntu 16.04.3 64-bit PC (AMD64)
, macOS Catalina (x86_64) and TensorFlow devel Docker image
[tensorflow/tensorflow:devel](https://hub.docker.com/r/tensorflow/tensorflow/tags/).
**Note:** This feature is available since version 2.4.
#### Prerequisites
You need CMake installed and a copy of the TensorFlow source code. Please check
[Build TensorFlow Lite with CMake](https://www.tensorflow.org/lite/guide/build_cmake)
page for the details.
To build the PIP package for your workstation, you can run the following
commands.
```sh
PYTHON=python3 tensorflow/lite/tools/pip_package/build_pip_package_with_cmake.sh native
```
**Note:** If you have multiple Python interpreters available, specify the exact
Python version with `PYTHON` variable. (Currently, it supports Python 3.7 or
higher)
## ARM cross compilation
For ARM cross compilation, it's recommended to use Docker since it makes easier
to setup cross build environment. Also you needs a `target` option to figure out
the target architecture.
There is a helper tool in Makefile `tensorflow/lite/tools/pip_package/Makefile`
available to invoke a build command using a pre-defined Docker container. On a
Docker host machine, you can run a build command as followings.
```sh
make -C tensorflow/lite/tools/pip_package docker-build \
TENSORFLOW_TARGET=<target> PYTHON_VERSION=<python3 version>
```
**Note:** Python version 3.7 or higher is supported.
### Available target names
`tensorflow/lite/tools/pip_package/build_pip_package_with_cmake.sh` script needs
a target name to figure out target architecture. Here is the list of supported
targets.
Target | Target architecture | Comments
--------- | -------------------- | --------
armhf | ARMv7 VFP with Neon | Compatible with Raspberry Pi 3 and 4
rpi0 | ARMv6 | Compatible with Raspberry Pi Zero
aarch64 | aarch64 (ARM 64-bit) | [Coral Mendel Linux 4.0](https://coral.ai/) <br/> Raspberry Pi with [Ubuntu Server 20.04.01 LTS 64-bit](https://ubuntu.com/download/raspberry-pi)
native | Your workstation | It builds with "-mnative" optimization
<default> | Your workstation | Default target
### Build examples
Here are some example commands you can use.
#### armhf target for Python 3.7
```sh
make -C tensorflow/lite/tools/pip_package docker-build \
TENSORFLOW_TARGET=armhf PYTHON_VERSION=3.7
```
#### aarch64 target for Python 3.8
```sh
make -C tensorflow/lite/tools/pip_package docker-build \
TENSORFLOW_TARGET=aarch64 PYTHON_VERSION=3.8
```
#### How to use a custom toolchain?
If the generated binaries are not compatible with your target, you need to use
your own toolchain or provide custom build flags. (Check
[this](https://www.tensorflow.org/lite/guide/build_cmake_arm#check_your_target_environment)
to understand your target environment) In that case, you need to modify
`tensorflow/lite/tools/cmake/download_toolchains.sh` to use your own toolchain.
The toolchain script defines the following two variables for the
`build_pip_package_with_cmake.sh` script.
Variable | Purpose | example
-------------- | ------------------------ | -------------------------------
`ARMCC_PREFIX` | defines toolchain prefix | arm-linux-gnueabihf-
`ARMCC_FLAGS` | compilation flags | -march=armv7-a -mfpu=neon-vfpv4
**Note:** `ARMCC_FLAGS` might need to contain Python library include path. See
the `download_toolchains.sh` for the reference.
tensorflow/lite/g3doc/guide/build_ios.md
+243
View File
@@ -0,0 +1,243 @@
# Build TensorFlow Lite for iOS
This document describes how to build TensorFlow Lite iOS library on your own.
Normally, you do not need to locally build TensorFlow Lite iOS library. If you
just want to use it, the easiest way is using the prebuilt stable or nightly
releases of the TensorFlow Lite CocoaPods. See [iOS quickstart](ios.md) for more
details on how to use them in your iOS projects.
## Building locally
In some cases, you might wish to use a local build of TensorFlow Lite, for
example when you want to make local changes to TensorFlow Lite and test those
changes in your iOS app or you prefer using static framework to our provided
dynamic one. To create a universal iOS framework for TensorFlow Lite locally,
you need to build it using Bazel on a macOS machine.
### Install Xcode
If you have not already, you will need to install Xcode 8 or later and the tools
using `xcode-select`:
```sh
xcode-select --install
```
If this is a new install, you will need to accept the license agreement for all
users with the following command:
```sh
sudo xcodebuild -license accept
```
### Install Bazel
Bazel is the primary build system for TensorFlow. Install Bazel as per the
[instructions on the Bazel website][bazel-install]. Make sure to choose a
version between `_TF_MIN_BAZEL_VERSION` and `_TF_MAX_BAZEL_VERSION` in
[`configure.py` file][configure-py] at the root of `tensorflow` repository.
### Configure WORKSPACE and .bazelrc
Run the `./configure` script in the root TensorFlow checkout directory, and
answer "Yes" when the script asks if you wish to build TensorFlow with iOS
support.
### Build TensorFlowLiteC dynamic framework (recommended)
Note: This step is not necessary if (1) you are using Bazel for your app, or (2)
you only want to test local changes to the Swift or Objective-C APIs. In these
cases, skip to the [Use in your own application](#use_in_your_own_application)
section below.
Once Bazel is properly configured with iOS support, you can build the
`TensorFlowLiteC` framework with the following command.
```sh
bazel build --config=ios_fat -c opt --cxxopt=--std=c++17 \
//tensorflow/lite/ios:TensorFlowLiteC_framework
```
This command will generate the `TensorFlowLiteC_framework.zip` file under
`bazel-bin/tensorflow/lite/ios/` directory under your TensorFlow root directory.
By default, the generated framework contains a "fat" binary, containing armv7,
arm64, and x86_64 (but no i386). To see the full list of build flags used when
you specify `--config=ios_fat`, please refer to the iOS configs section in the
[`.bazelrc` file][bazelrc].
### Build TensorFlowLiteC static framework
By default, we only distribute the dynamic framework via Cocoapods. If you want
to use the static framework instead, you can build the `TensorFlowLiteC` static
framework with the following command:
```
bazel build --config=ios_fat -c opt --cxxopt=--std=c++17 \
//tensorflow/lite/ios:TensorFlowLiteC_static_framework
```
The command will generate a file named `TensorFlowLiteC_static_framework.zip`
under `bazel-bin/tensorflow/lite/ios/` directory under your TensorFlow root
directory. This static framework can be used in the exact same way as the
dynamic one.
### Selectively build TFLite frameworks
You can build smaller frameworks targeting only a set of models using selective
build, which will skip unused operations in your model set and only include the
op kernels required to run the given set of models. The command is as following:
```sh
bash tensorflow/lite/ios/build_frameworks.sh \
--input_models=model1.tflite,model2.tflite \
--target_archs=x86_64,armv7,arm64
```
The above command will generate the static framework
`bazel-bin/tensorflow/lite/ios/tmp/TensorFlowLiteC_framework.zip` for TensorFlow
Lite built-in and custom ops; and optionally, generates the static framework
`bazel-bin/tensorflow/lite/ios/tmp/TensorFlowLiteSelectTfOps_framework.zip` if
your models contain Select TensorFlow ops. Note that the `--target_archs` flag
can be used to specify your deployment architectures.
## Use in your own application
### CocoaPods developers
There are three CocoaPods for TensorFlow Lite:
* `TensorFlowLiteSwift`: Provides the Swift APIs for TensorFlow Lite.
* `TensorFlowLiteObjC`: Provides the Objective-C APIs for TensorFlow Lite.
* `TensorFlowLiteC`: Common base pod, which embeds the TensorFlow Lite core
runtime and exposes the base C APIs used by the above two pods. Not meant to
be directly used by users.
As a developer, you should choose either `TensorFlowLiteSwift` or
`TensorFlowLiteObjC` pod based on the language in which your app is written, but
not both. The exact steps for using local builds of TensorFlow Lite differ,
depending on which exact part you would like to build.
#### Using local Swift or Objective-C APIs
If you are using CocoaPods, and only wish to test some local changes to the
TensorFlow Lite's [Swift APIs][swift-api] or [Objective-C APIs][objc-api],
follow the steps here.
1. Make changes to the Swift or Objective-C APIs in your `tensorflow` checkout.
1. Open the `TensorFlowLite(Swift|ObjC).podspec` file, and update this line: \
`s.dependency 'TensorFlowLiteC', "#{s.version}"` \
to be: \
`s.dependency 'TensorFlowLiteC', "~> 0.0.1-nightly"` \
This is to ensure that you are building your Swift or Objective-C APIs
against the latest available nightly version of `TensorFlowLiteC` APIs
(built every night between 1-4AM Pacific Time) rather than the stable
version, which may be outdated compared to your local `tensorflow` checkout.
Alternatively, you could choose to publish your own version of
`TensorFlowLiteC` and use that version (see
[Using local TensorFlow Lite core](#using_local_tensorflow_lite_core)
section below).
1. In the `Podfile` of your iOS project, change the dependency as follows to
point to the local path to your `tensorflow` root directory. \
For Swift: \
`pod 'TensorFlowLiteSwift', :path => '<your_tensorflow_root_dir>'` \
For Objective-C: \
`pod 'TensorFlowLiteObjC', :path => '<your_tensorflow_root_dir>'`
1. Update your pod installation from your iOS project root directory. \
`$ pod update`
1. Reopen the generated workspace (`<project>.xcworkspace`) and rebuild your
app within Xcode.
#### Using local TensorFlow Lite core
You can set up a private CocoaPods specs repository, and publish your custom
`TensorFlowLiteC` framework to your private repo. You can copy this
[podspec file][tflite-podspec] and modify a few values:
```ruby
...
s.version = <your_desired_version_tag>
...
# Note the `///`, two from the `file://` and one from the `/path`.
s.source = { :http => "file:///path/to/TensorFlowLiteC_framework.zip" }
...
s.vendored_frameworks = 'TensorFlowLiteC.framework'
...
```
After creating your own `TensorFlowLiteC.podspec` file, you can follow the
[instructions on using private CocoaPods][private-cocoapods] to use it in your
own project. You can also modify the `TensorFlowLite(Swift|ObjC).podspec` to
point to your custom `TensorFlowLiteC` pod and use either Swift or Objective-C
pod in your app project.
### Bazel developers
If you are using Bazel as the main build tool, you can simply add
`TensorFlowLite` dependency to your target in your `BUILD` file.
For Swift:
```python
swift_library(
deps = [
"//tensorflow/lite/swift:TensorFlowLite",
],
)
```
For Objective-C:
```python
objc_library(
deps = [
"//tensorflow/lite/objc:TensorFlowLite",
],
)
```
When you build your app project, any changes to the TensorFlow Lite library will
be picked up and built into your app.
### Modify Xcode project settings directly
It is highly recommended to use CocoaPods or Bazel for adding TensorFlow Lite
dependency into your project. If you still wish to add `TensorFlowLiteC`
framework manually, you'll need to add the `TensorFlowLiteC` framework as an
embedded framework to your application project. Unzip the
`TensorFlowLiteC_framework.zip` generated from the above build to get the
`TensorFlowLiteC.framework` directory. This directory is the actual framework
which Xcode can understand.
Once you've prepared the `TensorFlowLiteC.framework`, first you need to add it
as an embedded binary to your app target. The exact project settings section for
this may differ depending on your Xcode version.
* Xcode 11: Go to the 'General' tab of the project editor for your app target,
and add the `TensorFlowLiteC.framework` under 'Frameworks, Libraries, and
Embedded Content' section.
* Xcode 10 and below: Go to the 'General' tab of the project editor for your
app target, and add the `TensorFlowLiteC.framework` under 'Embedded
Binaries'. The framework should also be added automatically under 'Linked
Frameworks and Libraries' section.
When you add the framework as an embedded binary, Xcode would also update the
'Framework Search Paths' entry under 'Build Settings' tab to include the parent
directory of your framework. In case this does not happen automatically, you
should manually add the parent directory of the `TensorFlowLiteC.framework`
directory.
Once these two settings are done, you should be able to import and call the
TensorFlow Lite's C API, defined by the header files under
`TensorFlowLiteC.framework/Headers` directory.
[bazel-install]: https://docs.bazel.build/versions/master/install-os-x.html
[bazelrc]: https://github.com/tensorflow/tensorflow/blob/master/.bazelrc
[configure-py]: https://github.com/tensorflow/tensorflow/blob/master/configure.py
[objc-api]: https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/objc
[private-cocoapods]: https://guides.cocoapods.org/making/private-cocoapods.html
[swift-api]: https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/swift
[tflite-podspec]: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/ios/TensorFlowLiteC.podspec
tensorflow/lite/g3doc/guide/faq.md
+122
View File
@@ -0,0 +1,122 @@
# Frequently Asked Questions
If you don't find an answer to your question here, please look through our
detailed documentation for the topic or file a
[GitHub issue](https://github.com/tensorflow/tensorflow/issues).
## Model Conversion
#### What formats are supported for conversion from TensorFlow to TensorFlow Lite?
The supported formats are listed [here](../models/convert/index#python_api)
#### Why are some operations not implemented in TensorFlow Lite?
In order to keep TFLite lightweight, only certain TF operators (listed in the
[allowlist](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/guide/op_select_allowlist.md))
are supported in TFLite.
#### Why doesn't my model convert?
Since the number of TensorFlow Lite operations is smaller than TensorFlow's,
some models may not be able to convert. Some common errors are listed
[here](../models/convert/index#conversion-errors).
For conversion issues not related to missing operations or control flow ops,
search our
[GitHub issues](https://github.com/tensorflow/tensorflow/issues?q=label%3Acomp%3Alite+)
or file a [new one](https://github.com/tensorflow/tensorflow/issues).
#### How do I test that a TensorFlow Lite model behaves the same as the original TensorFlow model?
The best way to test is to compare the outputs of the TensorFlow and the
TensorFlow Lite models for the same inputs (test data or random inputs) as shown
[here](inference#load-and-run-a-model-in-python).
#### How do I determine the inputs/outputs for GraphDef protocol buffer?
The easiest way to inspect a graph from a `.pb` file is to use
[Netron](https://github.com/lutzroeder/netron), an open-source viewer for
machine learning models.
If Netron cannot open the graph, you can try the
[summarize_graph](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/tools/graph_transforms/README.md#inspecting-graphs)
tool.
If the summarize_graph tool yields an error, you can visualize the GraphDef with
[TensorBoard](https://www.tensorflow.org/guide/summaries_and_tensorboard) and
look for the inputs and outputs in the graph. To visualize a `.pb` file, use the
[`import_pb_to_tensorboard.py`](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/tools/import_pb_to_tensorboard.py)
script like below:
```shell
python import_pb_to_tensorboard.py --model_dir <model path> --log_dir <log dir path>
```
#### How do I inspect a `.tflite` file?
[Netron](https://github.com/lutzroeder/netron) is the easiest way to visualize a
TensorFlow Lite model.
If Netron cannot open your TensorFlow Lite model, you can try the
[visualize.py](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/tools/visualize.py)
script in our repository.
If you're using TF 2.5 or a later version
```shell
python -m tensorflow.lite.tools.visualize model.tflite visualized_model.html
```
Otherwise, you can run this script with Bazel
* [Clone the TensorFlow repository](https://www.tensorflow.org/install/source)
* Run the `visualize.py` script with bazel:
```shell
bazel run //tensorflow/lite/tools:visualize model.tflite visualized_model.html
```
## Optimization
#### How do I reduce the size of my converted TensorFlow Lite model?
[Post-training quantization](../performance/post_training_quantization) can
be used during conversion to TensorFlow Lite to reduce the size of the model.
Post-training quantization quantizes weights to 8-bits of precision from
floating-point and dequantizes them during runtime to perform floating point
computations. However, note that this could have some accuracy implications.
If retraining the model is an option, consider
[Quantization-aware training](https://github.com/tensorflow/tensorflow/tree/r1.13/tensorflow/contrib/quantize).
However, note that quantization-aware training is only available for a subset of
convolutional neural network architectures.
For a deeper understanding of different optimization methods, look at
[Model optimization](../performance/model_optimization).
#### How do I optimize TensorFlow Lite performance for my machine learning task?
The high-level process to optimize TensorFlow Lite performance looks something
like this:
* *Make sure that you have the right model for the task.* For image
classification, check out the
[TensorFlow Hub](https://tfhub.dev/s?deployment-format=lite&module-type=image-classification).
* *Tweak the number of threads.* Many TensorFlow Lite operators support
multi-threaded kernels. You can use `SetNumThreads()` in the
[C++ API](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/core/interpreter_builder.h#L110)
to do this. However, increasing threads results in performance variability
depending on the environment.
* *Use Hardware Accelerators.* TensorFlow Lite supports model acceleration for
specific hardware using delegates. See our
[Delegates](../performance/delegates) guide for information on what
accelerators are supported and how to use them with your model on-device.
* *(Advanced) Profile Model.* The Tensorflow Lite
[benchmarking tool](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/tools/benchmark)
has a built-in profiler that can show per-operator statistics. If you know
how you can optimize an operator’s performance for your specific platform,
you can implement a [custom operator](ops_custom).
For a more in-depth discussion on how to optimize performance, take a look at
[Best Practices](../performance/best_practices).
tensorflow/lite/g3doc/guide/index.md
+123
View File
@@ -0,0 +1,123 @@
# TensorFlow Lite
TensorFlow Lite is a set of tools that enables on-device machine learning by
helping developers run their models on mobile, embedded, and edge devices.
### Key features
- *Optimized for on-device machine learning*, by addressing 5 key constraints:
latency (there's no round-trip to a server), privacy (no personal data
leaves the device), connectivity (internet connectivity is not required),
size (reduced model and binary size) and power consumption (efficient
inference and a lack of network connections).
- *Multiple platform support*, covering [Android](../android) and [iOS](ios)
devices, [embedded Linux](python), and
[microcontrollers](../microcontrollers).
- *Diverse language support*, which includes Java, Swift, Objective-C, C++,
and Python.
- *High performance*, with [hardware acceleration](../performance/delegates)
and [model optimization](../performance/model_optimization).
- *End-to-end [examples](../examples)*, for common machine learning tasks such
as image classification, object detection, pose estimation, question
answering, text classification, etc. on multiple platforms.
Key Point: The TensorFlow Lite binary is ~1MB when all 125+ supported operators
are linked (for 32-bit ARM builds), and less than 300KB when using only the
operators needed for supporting the common image classification models
InceptionV3 and MobileNet.
## Development workflow
The following guide walks through each step of the workflow and provides links
to further instructions:
Note: Refer to the [performance best practices](../performance/best_practices)
guide for an ideal balance of performance, model size, and accuracy.
### 1. Generate a TensorFlow Lite model
A TensorFlow Lite model is represented in a special efficient portable format
known as [FlatBuffers](https://google.github.io/flatbuffers/){:.external}
(identified by the *.tflite* file extension). This provides several advantages
over TensorFlow's protocol buffer model format such as reduced size (small code
footprint) and faster inference (data is directly accessed without an extra
parsing/unpacking step) that enables TensorFlow Lite to execute efficiently on
devices with limited compute and memory resources.
A TensorFlow Lite model can optionally include *metadata* that has
human-readable model description and machine-readable data for automatic
generation of pre- and post-processing pipelines during on-device inference.
Refer to [Add metadata](../models/convert/metadata) for more details.
You can generate a TensorFlow Lite model in the following ways:
* **Use an existing TensorFlow Lite model:** Refer to
[TensorFlow Lite Examples](../examples) to pick an existing model. *Models
may or may not contain metadata.*
* **Create a TensorFlow Lite model:** Use the
[TensorFlow Lite Model Maker](../models/modify/model_maker) to create a
model with your own custom dataset. *By default, all models contain
metadata.*
* **Convert a TensorFlow model into a TensorFlow Lite model:** Use the
[TensorFlow Lite Converter](../models/convert/) to convert a TensorFlow
model into a TensorFlow Lite model. During conversion, you can apply
[optimizations](../performance/model_optimization) such as
[quantization](../performance/post_training_quantization) to reduce model
size and latency with minimal or no loss in accuracy. *By default, all
models don't contain metadata.*
### 2. Run Inference
*Inference* refers to the process of executing a TensorFlow Lite model on-device
to make predictions based on input data. You can run inference in the following
ways based on the model type:
* **Models *without* metadata**: Use the
[TensorFlow Lite Interpreter](inference) API. *Supported on multiple
platforms and languages such as Java, Swift, C++, Objective-C and Python.*
* **Models *with* metadata**: You can either leverage the out-of-box APIs
using the
[TensorFlow Lite Task Library](../inference_with_metadata/task_library/overview)
or build custom inference pipelines with the
[TensorFlow Lite Support Library](../inference_with_metadata/lite_support).
On android devices, users can automatically generate code wrappers using the
[Android Studio ML Model Binding](../inference_with_metadata/codegen#mlbinding)
or the
[TensorFlow Lite Code Generator](../inference_with_metadata/codegen#codegen).
*Supported only on Java (Android) while Swift (iOS) and C++ is work in
progress.*
On Android and iOS devices, you can improve performance using hardware
acceleration. On either platforms you can use a
[GPU Delegate](../performance/gpu), on android you can either use the
[NNAPI Delegate](../android/delegates/nnapi) (for newer devices) or the
[Hexagon Delegate](../android/delegates/hexagon) (on older devices) and on
iOS you can use the [Core ML Delegate](../performance/coreml_delegate). To add
support for new hardware accelerators, you can
[define your own delegate](../performance/implementing_delegate).
## Get started
You can refer to the following guides based on your target device:
* **Android and iOS:** Explore the [Android quickstart](../android/quickstart)
and [iOS quickstart](ios).
* **Embedded Linux:** Explore the [Python quickstart](python) for embedded
devices such as [Raspberry Pi](https://www.raspberrypi.org/){:.external} and
[Coral devices with Edge TPU](https://coral.withgoogle.com/){:.external}, or
C++ build instructions for [ARM](build_arm).
* **Microcontrollers:** Explore the
[TensorFlow Lite for Microcontrollers](../microcontrollers) library for
microcontrollers and DSPs that contain only a few kilobytes of memory.
## Technical constraints
* *All TensorFlow models* ***cannot*** *be converted into TensorFlow Lite
models*, refer to [Operator compatibility](ops_compatibility).
* *Unsupported on-device training*, however it is on our [Roadmap](roadmap).
tensorflow/lite/g3doc/guide/inference.md
+647
View File
@@ -0,0 +1,647 @@
# TensorFlow Lite inference
The term *inference* refers to the process of executing a TensorFlow Lite model
on-device in order to make predictions based on input data. To perform an
inference with a TensorFlow Lite model, you must run it through an
*interpreter*. The TensorFlow Lite interpreter is designed to be lean and fast.
The interpreter uses a static graph ordering and a custom (less-dynamic) memory
allocator to ensure minimal load, initialization, and execution latency.
This page describes how to access to the TensorFlow Lite interpreter and perform
an inference using C++, Java, and Python, plus links to other resources for each
[supported platform](#supported-platforms).
[TOC]
## Important concepts
TensorFlow Lite inference typically follows the following steps:
1. **Loading a model**
You must load the `.tflite` model into memory, which contains the model's
execution graph.
1. **Transforming data**
Raw input data for the model generally does not match the input data format
expected by the model. For example, you might need to resize an image or
change the image format to be compatible with the model.
1. **Running inference**
This step involves using the TensorFlow Lite API to execute the model. It
involves a few steps such as building the interpreter, and allocating
tensors, as described in the following sections.
1. **Interpreting output**
When you receive results from the model inference, you must interpret the
tensors in a meaningful way that's useful in your application.
For example, a model might return only a list of probabilities. It's up to
you to map the probabilities to relevant categories and present it to your
end-user.
## Supported platforms
TensorFlow inference APIs are provided for most common mobile/embedded platforms
such as [Android](#android-platform), [iOS](#ios-platform) and
[Linux](#linux-platform), in multiple programming languages.
In most cases, the API design reflects a preference for performance over ease of
use. TensorFlow Lite is designed for fast inference on small devices, so it
should be no surprise that the APIs try to avoid unnecessary copies at the
expense of convenience. Similarly, consistency with TensorFlow APIs was not an
explicit goal and some variance between languages is to be expected.
Across all libraries, the TensorFlow Lite API enables you to load models, feed
inputs, and retrieve inference outputs.
### Android Platform
On Android, TensorFlow Lite inference can be performed using either Java or C++
APIs. The Java APIs provide convenience and can be used directly within your
Android Activity classes. The C++ APIs offer more flexibility and speed, but may
require writing JNI wrappers to move data between Java and C++ layers.
See below for details about using [C++](#load-and-run-a-model-in-c) and
[Java](#load-and-run-a-model-in-java), or follow the
[Android quickstart](../android) for a tutorial and example code.
#### TensorFlow Lite Android wrapper code generator
Note: TensorFlow Lite wrapper code generator is in experimental (beta) phase and
it currently only supports Android.
For TensorFlow Lite model enhanced with [metadata](../inference_with_metadata/overview),
developers can use the TensorFlow Lite Android wrapper code generator to create
platform specific wrapper code. The wrapper code removes the need to interact
directly with `ByteBuffer` on Android. Instead, developers can interact with the
TensorFlow Lite model with typed objects such as `Bitmap` and `Rect`. For more
information, please refer to the
[TensorFlow Lite Android wrapper code generator](../inference_with_metadata/codegen.md).
### iOS Platform
On iOS, TensorFlow Lite is available with native iOS libraries written in
[Swift](https://www.tensorflow.org/code/tensorflow/lite/swift)
and
[Objective-C](https://www.tensorflow.org/code/tensorflow/lite/objc).
You can also use
[C API](https://www.tensorflow.org/code/tensorflow/lite/c/c_api.h)
directly in Objective-C codes.
See below for details about using [Swift](#load-and-run-a-model-in-swift),
[Objective-C](#load-and-run-a-model-in-objective-c) and the
[C API](#using-c-api-in-objective-c-code), or follow the
[iOS quickstart](ios.md) for a tutorial and example code.
### Linux Platform
On Linux platforms (including [Raspberry Pi](build_arm)), you can run
inferences using TensorFlow Lite APIs available in
[C++](#load-and-run-a-model-in-c) and [Python](#load-and-run-a-model-in-python),
as shown in the following sections.
## Running a model
Running a TensorFlow Lite model involves a few simple steps:
1. Load the model into memory.
2. Build an `Interpreter` based on an existing model.
3. Set input tensor values. (Optionally resize input tensors if the predefined
sizes are not desired.)
4. Invoke inference.
5. Read output tensor values.
Following sections describe how these steps can be done in each language.
## Load and run a model in Java
*Platform: Android*
The Java API for running an inference with TensorFlow Lite is primarily designed
for use with Android, so it's available as an Android library dependency:
`org.tensorflow:tensorflow-lite`.
In Java, you'll use the `Interpreter` class to load a model and drive model
inference. In many cases, this may be the only API you need.
You can initialize an `Interpreter` using a `.tflite` file:
```java
public Interpreter(@NotNull File modelFile);
```
Or with a `MappedByteBuffer`:
```java
public Interpreter(@NotNull MappedByteBuffer mappedByteBuffer);
```
In both cases, you must provide a valid TensorFlow Lite model or the API throws
`IllegalArgumentException`. If you use `MappedByteBuffer` to initialize an
`Interpreter`, it must remain unchanged for the whole lifetime of the
`Interpreter`.
The preferred way to run inference on a model is to use signatures -
Available for models converted starting Tensorflow 2.5
```Java
try (Interpreter interpreter = new Interpreter(file_of_tensorflowlite_model)) {
Map<String, Object> inputs = new HashMap<>();
inputs.put("input_1", input1);
inputs.put("input_2", input2);
Map<String, Object> outputs = new HashMap<>();
outputs.put("output_1", output1);
interpreter.runSignature(inputs, outputs, "mySignature");
}
```
The `runSignature` method takes three arguments:
- **Inputs** : map for inputs from input name in the signature to an input
object.
- **Outputs** : map for output mapping from output name in signature to output
data.
- **Signature Name** [optional]: Signature name (Can be left empty if the
model has single signature).
Another way to run an inference when the model doesn't
have a defined signatures.
Simply call `Interpreter.run()`. For example:
```java
try (Interpreter interpreter = new Interpreter(file_of_a_tensorflowlite_model)) {
interpreter.run(input, output);
}
```
The `run()` method takes only one input and returns only one output. So if your
model has multiple inputs or multiple outputs, instead use:
```java
interpreter.runForMultipleInputsOutputs(inputs, map_of_indices_to_outputs);
```
In this case, each entry in `inputs` corresponds to an input tensor and
`map_of_indices_to_outputs` maps indices of output tensors to the corresponding
output data.
In both cases, the tensor indices should correspond to the values you gave to
the [TensorFlow Lite Converter](../models/convert/) when you created the model. Be
aware that the order of tensors in `input` must match the order given to the
TensorFlow Lite Converter.
The `Interpreter` class also provides convenient functions for you to get the
index of any model input or output using an operation name:
```java
public int getInputIndex(String opName);
public int getOutputIndex(String opName);
```
If `opName` is not a valid operation in the model, it throws an
`IllegalArgumentException`.
Also beware that `Interpreter` owns resources. To avoid memory leak, the
resources must be released after use by:
```java
interpreter.close();
```
For an example project with Java, see the
[Android image classification sample](https://github.com/tensorflow/examples/tree/master/lite/examples/image_classification/android).
### Supported data types (in Java)
To use TensorFlow Lite, the data types of the input and output tensors must be
one of the following primitive types:
* `float`
* `int`
* `long`
* `byte`
`String` types are also supported, but they are encoded differently than the
primitive types. In particular, the shape of a string Tensor dictates the number
and arrangement of strings in the Tensor, with each element itself being a
variable length string. In this sense, the (byte) size of the Tensor cannot be
computed from the shape and type alone, and consequently strings cannot be
provided as a single, flat `ByteBuffer` argument. You can see some examples in
this [page](https://www.tensorflow.org/lite/api_docs/java/org/tensorflow/lite/Interpreter).
If other data types, including boxed types like `Integer` and `Float`, are used,
an `IllegalArgumentException` will be thrown.
#### Inputs
Each input should be an array or multi-dimensional array of the supported
primitive types, or a raw `ByteBuffer` of the appropriate size. If the input is
an array or multi-dimensional array, the associated input tensor will be
implicitly resized to the array's dimensions at inference time. If the input is
a ByteBuffer, the caller should first manually resize the associated input
tensor (via `Interpreter.resizeInput()`) before running inference.
When using `ByteBuffer`, prefer using direct byte buffers, as this allows the
`Interpreter` to avoid unnecessary copies. If the `ByteBuffer` is a direct byte
buffer, its order must be `ByteOrder.nativeOrder()`. After it is used for a
model inference, it must remain unchanged until the model inference is finished.
#### Outputs
Each output should be an array or multi-dimensional array of the supported
primitive types, or a ByteBuffer of the appropriate size. Note that some models
have dynamic outputs, where the shape of output tensors can vary depending on
the input. There's no straightforward way of handling this with the existing
Java inference API, but planned extensions will make this possible.
## Load and run a model in Swift
*Platform: iOS*
The
[Swift API](https://www.tensorflow.org/code/tensorflow/lite/swift)
is available in `TensorFlowLiteSwift` Pod from Cocoapods.
First, you need to import `TensorFlowLite` module.
```swift
import TensorFlowLite
```
```swift
// Getting model path
guard
let modelPath = Bundle.main.path(forResource: "model", ofType: "tflite")
else {
// Error handling...
}
do {
// Initialize an interpreter with the model.
let interpreter = try Interpreter(modelPath: modelPath)
// Allocate memory for the model's input `Tensor`s.
try interpreter.allocateTensors()
let inputData: Data // Should be initialized
// input data preparation...
// Copy the input data to the input `Tensor`.
try self.interpreter.copy(inputData, toInputAt: 0)
// Run inference by invoking the `Interpreter`.
try self.interpreter.invoke()
// Get the output `Tensor`
let outputTensor = try self.interpreter.output(at: 0)
// Copy output to `Data` to process the inference results.
let outputSize = outputTensor.shape.dimensions.reduce(1, {x, y in x * y})
let outputData =
UnsafeMutableBufferPointer<Float32>.allocate(capacity: outputSize)
outputTensor.data.copyBytes(to: outputData)
if (error != nil) { /* Error handling... */ }
} catch error {
// Error handling...
}
```
## Load and run a model in Objective-C
*Platform: iOS*
The
[Objective-C API](https://www.tensorflow.org/code/tensorflow/lite/objc)
is available in `TensorFlowLiteObjC` Pod from Cocoapods.
First, you need to import `TensorFlowLite` module.
```objc
@import TensorFlowLite;
```
```objc
NSString *modelPath = [[NSBundle mainBundle] pathForResource:@"model"
ofType:@"tflite"];
NSError *error;
// Initialize an interpreter with the model.
TFLInterpreter *interpreter = [[TFLInterpreter alloc] initWithModelPath:modelPath
error:&error];
if (error != nil) { /* Error handling... */ }
// Allocate memory for the model's input `TFLTensor`s.
[interpreter allocateTensorsWithError:&error];
if (error != nil) { /* Error handling... */ }
NSMutableData *inputData; // Should be initialized
// input data preparation...
// Get the input `TFLTensor`
TFLTensor *inputTensor = [interpreter inputTensorAtIndex:0 error:&error];
if (error != nil) { /* Error handling... */ }
// Copy the input data to the input `TFLTensor`.
[inputTensor copyData:inputData error:&error];
if (error != nil) { /* Error handling... */ }
// Run inference by invoking the `TFLInterpreter`.
[interpreter invokeWithError:&error];
if (error != nil) { /* Error handling... */ }
// Get the output `TFLTensor`
TFLTensor *outputTensor = [interpreter outputTensorAtIndex:0 error:&error];
if (error != nil) { /* Error handling... */ }
// Copy output to `NSData` to process the inference results.
NSData *outputData = [outputTensor dataWithError:&error];
if (error != nil) { /* Error handling... */ }
```
### Using C API in Objective-C code
Currently Objective-C API does not support delegates. In order to use delegates
with Objective-C code, you need to directly call underlying
[C API](https://www.tensorflow.org/code/tensorflow/lite/c/c_api.h).
```c
#include "tensorflow/lite/c/c_api.h"
```
```c
TfLiteModel* model = TfLiteModelCreateFromFile([modelPath UTF8String]);
TfLiteInterpreterOptions* options = TfLiteInterpreterOptionsCreate();
// Create the interpreter.
TfLiteInterpreter* interpreter = TfLiteInterpreterCreate(model, options);
// Allocate tensors and populate the input tensor data.
TfLiteInterpreterAllocateTensors(interpreter);
TfLiteTensor* input_tensor =
TfLiteInterpreterGetInputTensor(interpreter, 0);
TfLiteTensorCopyFromBuffer(input_tensor, input.data(),
input.size() * sizeof(float));
// Execute inference.
TfLiteInterpreterInvoke(interpreter);
// Extract the output tensor data.
const TfLiteTensor* output_tensor =
TfLiteInterpreterGetOutputTensor(interpreter, 0);
TfLiteTensorCopyToBuffer(output_tensor, output.data(),
output.size() * sizeof(float));
// Dispose of the model and interpreter objects.
TfLiteInterpreterDelete(interpreter);
TfLiteInterpreterOptionsDelete(options);
TfLiteModelDelete(model);
```
## Load and run a model in C++
*Platforms: Android, iOS, and Linux*
Note: C++ API on iOS is only available when using bazel.
In C++, the model is stored in
[`FlatBufferModel`](https://www.tensorflow.org/lite/api_docs/cc/class/tflite/flat-buffer-model.html)
class. It encapsulates a TensorFlow Lite model and you can build it in a couple
of different ways, depending on where the model is stored:
```c++
class FlatBufferModel {
 // Build a model based on a file. Return a nullptr in case of failure.
 static std::unique_ptr<FlatBufferModel> BuildFromFile(
     const char* filename,
     ErrorReporter* error_reporter);
 // Build a model based on a pre-loaded flatbuffer. The caller retains
 // ownership of the buffer and should keep it alive until the returned object
 // is destroyed. Return a nullptr in case of failure.
 static std::unique_ptr<FlatBufferModel> BuildFromBuffer(
     const char* buffer,
     size_t buffer_size,
     ErrorReporter* error_reporter);
};
```
Note: If TensorFlow Lite detects the presence of the
[Android NNAPI](https://developer.android.com/ndk/guides/neuralnetworks), it
will automatically try to use shared memory to store the `FlatBufferModel`.
Now that you have the model as a `FlatBufferModel` object, you can execute it
with an
[`Interpreter`](https://www.tensorflow.org/lite/api_docs/cc/class/tflite/interpreter.html).
A single `FlatBufferModel` can be used simultaneously by more than one
`Interpreter`.
Caution: The `FlatBufferModel` object must remain valid until all instances of
`Interpreter` using it have been destroyed.
The important parts of the `Interpreter` API are shown in the code snippet
below. It should be noted that:
* Tensors are represented by integers, in order to avoid string comparisons
(and any fixed dependency on string libraries).
* An interpreter must not be accessed from concurrent threads.
* Memory allocation for input and output tensors must be triggered by calling
`AllocateTensors()` right after resizing tensors.
The simplest usage of TensorFlow Lite with C++ looks like this:
```c++
// Load the model
std::unique_ptr<tflite::FlatBufferModel> model =
tflite::FlatBufferModel::BuildFromFile(filename);
// Build the interpreter
tflite::ops::builtin::BuiltinOpResolver resolver;
std::unique_ptr<tflite::Interpreter> interpreter;
tflite::InterpreterBuilder(*model, resolver)(&interpreter);
// Resize input tensors, if desired.
interpreter->AllocateTensors();
float* input = interpreter->typed_input_tensor<float>(0);
// Fill `input`.
interpreter->Invoke();
float* output = interpreter->typed_output_tensor<float>(0);
```
For more example code, see
[`minimal.cc`](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/examples/minimal/minimal.cc)
and
[`label_image.cc`](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/examples/label_image/label_image.cc).
## Load and run a model in Python
*Platform: Linux*
The Python API for running an inference is provided in the `tf.lite` module.
From which, you mostly need only
[`tf.lite.Interpreter`](https://www.tensorflow.org/api_docs/python/tf/lite/Interpreter)
to load a model and run an inference.
The following example shows how to use the Python interpreter to load a
`.tflite` file and run inference with random input data:
This example is recommended if you're converting from SavedModel with a defined
SignatureDef.
Available starting from TensorFlow 2.5
```python
class TestModel(tf.Module):
def __init__(self):
super(TestModel, self).__init__()
@tf.function(input_signature=[tf.TensorSpec(shape=[1, 10], dtype=tf.float32)])
def add(self, x):
'''
Simple method that accepts single input 'x' and returns 'x' + 4.
'''
# Name the output 'result' for convenience.
return {'result' : x + 4}
SAVED_MODEL_PATH = 'content/saved_models/test_variable'
TFLITE_FILE_PATH = 'content/test_variable.tflite'
# Save the model
module = TestModel()
# You can omit the signatures argument and a default signature name will be
# created with name 'serving_default'.
tf.saved_model.save(
module, SAVED_MODEL_PATH,
signatures={'my_signature':module.add.get_concrete_function()})
# Convert the model using TFLiteConverter
converter = tf.lite.TFLiteConverter.from_saved_model(SAVED_MODEL_PATH)
tflite_model = converter.convert()
with open(TFLITE_FILE_PATH, 'wb') as f:
f.write(tflite_model)
# Load the TFLite model in TFLite Interpreter
interpreter = tf.lite.Interpreter(TFLITE_FILE_PATH)
# There is only 1 signature defined in the model,
# so it will return it by default.
# If there are multiple signatures then we can pass the name.
my_signature = interpreter.get_signature_runner()
# my_signature is callable with input as arguments.
output = my_signature(x=tf.constant([1.0], shape=(1,10), dtype=tf.float32))
# 'output' is dictionary with all outputs from the inference.
# In this case we have single output 'result'.
print(output['result'])
```
Another example if the model doesn't have SignatureDefs defined.
```python
import numpy as np
import tensorflow as tf
# Load the TFLite model and allocate tensors.
interpreter = tf.lite.Interpreter(model_path="converted_model.tflite")
interpreter.allocate_tensors()
# Get input and output tensors.
input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()
# Test the model on random input data.
input_shape = input_details[0]['shape']
input_data = np.array(np.random.random_sample(input_shape), dtype=np.float32)
interpreter.set_tensor(input_details[0]['index'], input_data)
interpreter.invoke()
# The function `get_tensor()` returns a copy of the tensor data.
# Use `tensor()` in order to get a pointer to the tensor.
output_data = interpreter.get_tensor(output_details[0]['index'])
print(output_data)
```
As an alternative to loading the model as a pre-converted `.tflite` file, you
can combine your code with the
[TensorFlow Lite Converter Python API](https://www.tensorflow.org/lite/api_docs/python/tf/lite/TFLiteConverter)
(`tf.lite.TFLiteConverter`), allowing you to convert your Keras model into the
TensorFlow Lite format and then run inference:
```python
import numpy as np
import tensorflow as tf
img = tf.keras.Input(shape=(64, 64, 3), name="img")
const = tf.constant([1., 2., 3.]) + tf.constant([1., 4., 4.])
val = img + const
out = tf.identity(val, name="out")
# Convert to TF Lite format
converter = tf.lite.TFLiteConverter.from_keras_model(tf.keras.models.Model(inputs=[img], outputs=[out]))
tflite_model = converter.convert()
# Load the TFLite model and allocate tensors.
interpreter = tf.lite.Interpreter(model_content=tflite_model)
interpreter.allocate_tensors()
# Continue to get tensors and so forth, as shown above...
```
For more Python sample code, see
[`label_image.py`](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/examples/python/label_image.py).
Tip: Run `help(tf.lite.Interpreter)` in the Python terminal to get detailed
documentation about the interpreter.
## Run inference with dynamic shape model
If you want to run a model with dynamic input shape,
*resize the input shape* before running inference.
Otherwise, the `None` shape in Tensorflow models will be replaced by a
placeholder of `1` in TFLite models.
The following examples show how to resize the input shape before
running inference in different languages.
All the examples assume that the input shape is defined as `[1/None, 10]`, and
need to be resized to `[3, 10]`.
C++ example:
```c++
// Resize input tensors before allocate tensors
interpreter->ResizeInputTensor(/*tensor_index=*/0, std::vector<int>{3,10});
interpreter->AllocateTensors();
```
Python example:
```python
# Load the TFLite model in TFLite Interpreter
interpreter = tf.lite.Interpreter(model_path=TFLITE_FILE_PATH)
# Resize input shape for dynamic shape model and allocate tensor
interpreter.resize_tensor_input(interpreter.get_input_details()[0]['index'], [3, 10])
interpreter.allocate_tensors()
# Get input and output tensors.
input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()
```
</section>
## Supported operations
TensorFlow Lite supports a subset of TensorFlow operations with some
limitations. For full list of operations and limitations see
[TF Lite Ops page](https://www.tensorflow.org/mlir/tfl_ops).
tensorflow/lite/g3doc/guide/ios.md
+161
View File
@@ -0,0 +1,161 @@
# iOS quickstart
To get started with TensorFlow Lite on iOS, we recommend exploring the following
example:
<a class="button button-primary" href="https://github.com/tensorflow/examples/tree/master/lite/examples/image_classification/ios">iOS
image classification example</a>
For an explanation of the source code, you should also read
[TensorFlow Lite iOS image classification](https://github.com/tensorflow/examples/blob/master/lite/examples/image_classification/ios/README.md).
This example app uses
[image classification](https://www.tensorflow.org/lite/examples/image_classification/overview)
to continuously classify whatever it sees from the device's rear-facing camera,
displaying the top most probable classifications. It allows the user to choose
between a floating point or
[quantized](https://www.tensorflow.org/lite/performance/post_training_quantization)
model and select the number of threads to perform inference on.
Note: Additional iOS applications demonstrating TensorFlow Lite in a variety of
use cases are available in [Examples](https://www.tensorflow.org/lite/examples).
## Add TensorFlow Lite to your Swift or Objective-C project
TensorFlow Lite offers native iOS libraries written in
[Swift](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/swift)
and
[Objective-C](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/objc).
Start writing your own iOS code using the
[Swift image classification example](https://github.com/tensorflow/examples/tree/master/lite/examples/image_classification/ios)
as a starting point.
The sections below demonstrate how to add TensorFlow Lite Swift or Objective-C
to your project:
### CocoaPods developers
In your `Podfile`, add the TensorFlow Lite pod. Then, run `pod install`.
#### Swift
```ruby
use_frameworks!
pod 'TensorFlowLiteSwift'
```
#### Objective-C
```ruby
pod 'TensorFlowLiteObjC'
```
#### Specifying versions
There are stable releases, and nightly releases available for both
`TensorFlowLiteSwift` and `TensorFlowLiteObjC` pods. If you do not specify a
version constraint as in the above examples, CocoaPods will pull the latest
stable release by default.
You can also specify a version constraint. For example, if you wish to depend on
version 2.10.0, you can write the dependency as:
```ruby
pod 'TensorFlowLiteSwift', '~> 2.10.0'
```
This will ensure the latest available 2.x.y version of the `TensorFlowLiteSwift`
pod is used in your app. Alternatively, if you want to depend on the nightly
builds, you can write:
```ruby
pod 'TensorFlowLiteSwift', '~> 0.0.1-nightly'
```
From 2.4.0 version and latest nightly releases, by default
[GPU](https://www.tensorflow.org/lite/performance/gpu) and
[Core ML delegates](https://www.tensorflow.org/lite/performance/coreml_delegate)
are excluded from the pod to reduce the binary size. You can include them by
specifying subspec:
```ruby
pod 'TensorFlowLiteSwift', '~> 0.0.1-nightly', :subspecs => ['CoreML', 'Metal']
```
This will allow you to use the latest features added to TensorFlow Lite. Note
that once the `Podfile.lock` file is created when you run `pod install` command
for the first time, the nightly library version will be locked at the current
date's version. If you wish to update the nightly library to the newer one, you
should run `pod update` command.
For more information on different ways of specifying version constraints, see
[Specifying pod versions](https://guides.cocoapods.org/using/the-podfile.html#specifying-pod-versions).
### Bazel developers
In your `BUILD` file, add the `TensorFlowLite` dependency to your target.
#### Swift
```python
swift_library(
deps = [
"//tensorflow/lite/swift:TensorFlowLite",
],
)
```
#### Objective-C
```python
objc_library(
deps = [
"//tensorflow/lite/objc:TensorFlowLite",
],
)
```
#### C/C++ API
Alternatively, you can use
[C API](https://www.tensorflow.org/code/tensorflow/lite/c/c_api.h)
or [C++ API](https://tensorflow.org/lite/api_docs/cc)
```python
# Using C API directly
objc_library(
deps = [
"//tensorflow/lite/c:c_api",
],
)
# Using C++ API directly
objc_library(
deps = [
"//tensorflow/lite:framework",
],
)
```
### Import the library
For Swift files, import the TensorFlow Lite module:
```swift
import TensorFlowLite
```
For Objective-C files, import the umbrella header:
```objectivec
#import "TFLTensorFlowLite.h"
```
Or, the module if you set `CLANG_ENABLE_MODULES = YES` in your Xcode project:
```objectivec
@import TFLTensorFlowLite;
```
Note: For CocoaPods developers who want to import the Objective-C TensorFlow
Lite module, you must also include `use_frameworks!` in your `Podfile`.
tensorflow/lite/g3doc/guide/model_analyzer.ipynb
+250
View File
@@ -0,0 +1,250 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "g_nWetWWd_ns"
},
"source": [
"##### Copyright 2024 The AI Edge Authors."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "2pHVBk_seED1"
},
"outputs": [],
"source": [
"#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "M7vSdG6sAIQn"
},
"source": [
"# TensorFlow Lite Model Analyzer"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "fwc5GKHBASdc"
},
"source": [
"<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
" <td>\n",
" <a target=\"_blank\" href=\"https://www.tensorflow.org/lite/guide/model_analyzer\"><img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" />View on TensorFlow.org</a>\n",
" </td>\n",
" <td>\n",
" <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/guide/model_analyzer.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
" </td>\n",
" <td>\n",
" <a target=\"_blank\" href=\"https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/guide/model_analyzer.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://storage.googleapis.com/tensorflow_docs/tensorflow/tensorflow/lite/g3doc/guide/model_analyzer.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\" />Download notebook</a>\n",
" </td>\n",
"</table>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "9ee074e4"
},
"source": [
"TensorFlow Lite Model Analyzer API helps you analyze models in TensorFlow Lite format by listing a model's structure.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JKwW0VfDKMWS"
},
"source": [
"## Model Analyzer API\n",
"\n",
"The following API is available for the TensorFlow Lite Model Analyzer.\n",
"\n",
"```\n",
"tf.lite.experimental.Analyzer.analyze(model_path=None,\n",
" model_content=None,\n",
" gpu_compatibility=False)\n",
"```\n",
"\n",
"You can find the API details from https://www.tensorflow.org/api_docs/python/tf/lite/experimental/Analyzer or run `help(tf.lite.experimental.Analyzer.analyze)` from a Python terminal.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "qi8Vk4_065jN"
},
"source": [
"## Basic usage with simple Keras model\n",
"\n",
"The following code shows basic usage of Model Analyzer. It shows contents of the converted Keras model in TFLite model content, formatted as a flatbuffer object."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "_jkg6UNtdz8c"
},
"outputs": [],
"source": [
"import tensorflow as tf\n",
"\n",
"model = tf.keras.models.Sequential([\n",
" tf.keras.layers.Flatten(input_shape=(128, 128)),\n",
" tf.keras.layers.Dense(256, activation='relu'),\n",
" tf.keras.layers.Dropout(0.2),\n",
" tf.keras.layers.Dense(10)\n",
"])\n",
"\n",
"fb_model = tf.lite.TFLiteConverter.from_keras_model(model).convert()\n",
"\n",
"tf.lite.experimental.Analyzer.analyze(model_content=fb_model)\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "pe_ZU5Zy7PeH"
},
"source": [
"## Basic usage with MobileNetV3Large Keras model\n",
"\n",
"This API works with large models such as MobileNetV3Large. Since the output is large, you might want to browse it with your favorite text editor."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "QFywJ_g56VW5"
},
"outputs": [],
"source": [
"model = tf.keras.applications.MobileNetV3Large()\n",
"fb_model = tf.lite.TFLiteConverter.from_keras_model(model).convert()\n",
"\n",
"tf.lite.experimental.Analyzer.analyze(model_content=fb_model)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "4BGqG2j9yqRf"
},
"source": [
"## Check GPU delegate compatibility\n",
"\n",
"The ModelAnalyzer API provides a way to check the [GPU delegate](https://www.tensorflow.org/lite/performance/gpu) compatibility of the given model by providing `gpu_compatibility=True` option.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "sVGC1oX33RkV"
},
"source": [
"### Case 1: When model is incompatibile\n",
"\n",
"The following code shows a way to use `gpu_compatibility=True` option for simple tf.function which uses `tf.slice` with a 2D tensor and `tf.cosh` which are not compatible with GPU delegate.\n",
"\n",
"You will see `GPU COMPATIBILITY WARNING` per every node which has compatibility issue(s)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "9GEg5plIzD-3"
},
"outputs": [],
"source": [
"import tensorflow as tf\n",
"\n",
"@tf.function(input_signature=[\n",
" tf.TensorSpec(shape=[4, 4], dtype=tf.float32)\n",
"])\n",
"def func(x):\n",
" return tf.cosh(x) + tf.slice(x, [1, 1], [1, 1])\n",
"\n",
"converter = tf.lite.TFLiteConverter.from_concrete_functions(\n",
" [func.get_concrete_function()], func)\n",
"converter.target_spec.supported_ops = [\n",
" tf.lite.OpsSet.TFLITE_BUILTINS,\n",
" tf.lite.OpsSet.SELECT_TF_OPS,\n",
"]\n",
"fb_model = converter.convert()\n",
"\n",
"tf.lite.experimental.Analyzer.analyze(model_content=fb_model, gpu_compatibility=True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "BFU7HYb_2a8M"
},
"source": [
"### Case 2: When model is compatibile\n",
"\n",
"In this example, the given model is compatbile with GPU delegate.\n",
"\n",
"**Note:** Even though the tool doesn't find any compatibility issue, it doesn't guarantee that your model works well with GPU delegate on every device. There could be some runtime incompatibililty happen such as missing `CL_DEVICE_IMAGE_SUPPORT` feature by target OpenGL backend.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "85RgG6tQ3ABT"
},
"outputs": [],
"source": [
"model = tf.keras.models.Sequential([\n",
" tf.keras.layers.Flatten(input_shape=(128, 128)),\n",
" tf.keras.layers.Dense(256, activation='relu'),\n",
" tf.keras.layers.Dropout(0.2),\n",
" tf.keras.layers.Dense(10)\n",
"])\n",
"\n",
"fb_model = tf.lite.TFLiteConverter.from_keras_model(model).convert()\n",
"\n",
"tf.lite.experimental.Analyzer.analyze(model_content=fb_model, gpu_compatibility=True)"
]
}
],
"metadata": {
"colab": {
"collapsed_sections": [],
"name": "model_analyzer.ipynb",
"provenance": [],
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
tensorflow/lite/g3doc/guide/op_select_allowlist.md
+917
View File
@@ -0,0 +1,917 @@
# Supported Select TensorFlow operators
Caution: The operators list is updated frequently.
## TensorFlow core operators
The following is an exhaustive list of TensorFlow core operations that are
supported by TensorFlow Lite runtime with the Select TensorFlow Ops feature.
* `raw_ops.Abort`
* `raw_ops.Abs`
* `raw_ops.Add`
* `raw_ops.AddN`
* `raw_ops.AddV2`
* `raw_ops.AdjustContrast`
* `raw_ops.AdjustContrastv2`
* `raw_ops.AdjustHue`
* `raw_ops.AdjustSaturation`
* `raw_ops.All`
* `raw_ops.Angle`
* `raw_ops.Any`
* `raw_ops.ApplyAdadelta`
* `raw_ops.ApplyAdagrad`
* `raw_ops.ApplyAdagradDA`
* `raw_ops.ApplyAdagradV2`
* `raw_ops.ApplyAdam`
* `raw_ops.ApplyAdaMax`
* `raw_ops.ApplyAddSign`
* `raw_ops.ApplyCenteredRMSProp`
* `raw_ops.ApplyFtrl`
* `raw_ops.ApplyFtrlV2`
* `raw_ops.ApplyGradientDescent`
* `raw_ops.ApplyMomentum`
* `raw_ops.ApplyPowerSign`
* `raw_ops.ApplyProximalAdagrad`
* `raw_ops.ApplyProximalGradientDescent`
* `raw_ops.ApplyRMSProp`
* `raw_ops.ApproximateEqual`
* `raw_ops.ArgMax`
* `raw_ops.ArgMin`
* `raw_ops.AsString`
* `raw_ops.Assert`
* `raw_ops.Assign`
* `raw_ops.AssignAdd`
* `raw_ops.AssignAddVariableOp`
* `raw_ops.AssignSub`
* `raw_ops.AssignSubVariableOp`
* `raw_ops.AssignVariableOp`
* `raw_ops.Atan`
* `raw_ops.Atan2`
* `raw_ops.AudioSpectrogram`
* `raw_ops.AvgPool`
* `raw_ops.AvgPool3D`
* `raw_ops.AvgPool3DGrad`
* `raw_ops.AvgPoolGrad`
* `raw_ops.BatchCholesky`
* `raw_ops.BatchDatasetV2`
* `raw_ops.BatchMatMul`
* `raw_ops.BatchMatMulV2`
* `raw_ops.BatchMatrixBandPart`
* `raw_ops.BatchMatrixDiag`
* `raw_ops.BatchMatrixDiagPart`
* `raw_ops.BatchMatrixInverse`
* `raw_ops.BatchMatrixSetDiag`
* `raw_ops.BatchMatrixTriangularSolve`
* `raw_ops.BatchNormWithGlobalNormalization`
* `raw_ops.BatchNormWithGlobalNormalizationGrad`
* `raw_ops.BatchToSpace`
* `raw_ops.BatchToSpaceND`
* `raw_ops.BiasAdd`
* `raw_ops.BiasAddGrad`
* `raw_ops.BiasAddV1`
* `raw_ops.Bincount`
* `raw_ops.Bitcast`
* `raw_ops.BitwiseAnd`
* `raw_ops.BitwiseOr`
* `raw_ops.BitwiseXor`
* `raw_ops.BroadcastArgs`
* `raw_ops.BroadcastGradientArgs`
* `raw_ops.BroadcastTo`
* `raw_ops.Bucketize`
* `raw_ops.CTCBeamSearchDecoder`
* `raw_ops.CTCGreedyDecoder`
* `raw_ops.Case`
* `raw_ops.Cast`
* `raw_ops.Ceil`
* `raw_ops.CheckNumerics`
* `raw_ops.CheckNumericsV2`
* `raw_ops.Cholesky`
* `raw_ops.ClipByValue`
* `raw_ops.CombinedNonMaxSuppression`
* `raw_ops.Complex`
* `raw_ops.ComplexAbs`
* `raw_ops.Concat`
* `raw_ops.ConcatOffset`
* `raw_ops.ConcatV2`
* `raw_ops.Conj`
* `raw_ops.ConjugateTranspose`
* `raw_ops.Const`
* `raw_ops.ControlTrigger`
* `raw_ops.Conv2D`
* `raw_ops.Conv2DBackpropFilter`
* `raw_ops.Conv2DBackpropInput`
* `raw_ops.Conv3D`
* `raw_ops.Conv3DBackpropFilter`
* `raw_ops.Conv3DBackpropFilterV2`
* `raw_ops.Conv3DBackpropInput`
* `raw_ops.Conv3DBackpropInputV2`
* `raw_ops.Cos`
* `raw_ops.Cosh`
* `raw_ops.CropAndResize`
* `raw_ops.CropAndResizeGradBoxes`
* `raw_ops.CropAndResizeGradImage`
* `raw_ops.CTCBeamSearchDecoder`
* `raw_ops.CTCGreedyDecoder`
* `raw_ops.Cumprod`
* `raw_ops.Cumsum`
* `raw_ops.CumulativeLogsumexp`
* `raw_ops.DataFormatDimMap`
* `raw_ops.DataFormatVecPermute`
* `raw_ops.DebugGradientIdentity`
* `raw_ops.DebugGradientRefIdentity`
* `raw_ops.DecodeAndCropJpeg`
* `raw_ops.DecodeBase64`
* `raw_ops.DecodeBmp`
* `raw_ops.DecodeGif`
* `raw_ops.DecodeImage`
* `raw_ops.DecodeJpeg`
* `raw_ops.DecodePaddedRaw`
* `raw_ops.DecodePng`
* `raw_ops.DecodeRaw`
* `raw_ops.DecodeWav`
* `raw_ops.DeepCopy`
* `raw_ops.DeleteSessionTensor`
* `raw_ops.DenseBincount`
* `raw_ops.DenseToDenseSetOperation`
* `raw_ops.DenseToSparseSetOperation`
* `raw_ops.DepthToSpace`
* `raw_ops.DepthwiseConv2dNative`
* `raw_ops.DepthwiseConv2dNativeBackpropFilter`
* `raw_ops.DepthwiseConv2dNativeBackpropInput`
* `raw_ops.Dequantize`
* `raw_ops.DestroyResourceOp`
* `raw_ops.DestroyTemporaryVariable`
* `raw_ops.Diag`
* `raw_ops.DiagPart`
* `raw_ops.Dilation2D`
* `raw_ops.Dilation2DBackpropFilter`
* `raw_ops.Dilation2DBackpropInput`
* `raw_ops.Div`
* `raw_ops.DivNoNan`
* `raw_ops.DynamicPartition`
* `raw_ops.DynamicStitch`
* `raw_ops.Einsum`
* `raw_ops.Elu`
* `raw_ops.EluGrad`
* `raw_ops.Empty`
* `raw_ops.EmptyTensorList`
* `raw_ops.EmptyTensorMap`
* `raw_ops.EncodeBase64`
* `raw_ops.EncodeJpeg`
* `raw_ops.EncodeJpegVariableQuality`
* `raw_ops.EncodePng`
* `raw_ops.EncodeWav`
* `raw_ops.EnsureShape`
* `raw_ops.Enter`
* `raw_ops.Equal`
* `raw_ops.Erf`
* `raw_ops.Exit`
* `raw_ops.Exp`
* `raw_ops.ExpandDims`
* `raw_ops.ExtractImagePatches`
* `raw_ops.FakeQuantWithMinMaxArgs`
* `raw_ops.FakeQuantWithMinMaxArgsGradient`
* `raw_ops.FakeQuantWithMinMaxVars`
* `raw_ops.FakeQuantWithMinMaxVarsGradient`
* `raw_ops.FakeQuantWithMinMaxVarsPerChannel`
* `raw_ops.FakeQuantWithMinMaxVarsPerChannelGradient`
* `raw_ops.FakeQueue`
* `raw_ops.FFT`
* `raw_ops.FFT2D`
* `raw_ops.FFT3D`
* `raw_ops.FIFOQueue`
* `raw_ops.FIFOQueueV2`
* `raw_ops.Fill`
* `raw_ops.FilterDataset`
* `raw_ops.FinalizeDataset`
* `raw_ops.Fingerprint`
* `raw_ops.FlatMapDataset`
* `raw_ops.Floor`
* `raw_ops.FloorDiv`
* `raw_ops.FloorMod`
* `raw_ops.FusedBatchNorm`
* `raw_ops.FusedBatchNormGrad`
* `raw_ops.FusedBatchNormGradV2`
* `raw_ops.FusedBatchNormGradV3`
* `raw_ops.FusedBatchNormV2`
* `raw_ops.FusedBatchNormV3`
* `raw_ops.FusedPadConv2D`
* `raw_ops.FusedResizeAndPadConv2D`
* `raw_ops.Gather`
* `raw_ops.GatherNd`
* `raw_ops.GatherV2`
* `raw_ops.GetSessionHandle`
* `raw_ops.GetSessionHandleV2`
* `raw_ops.GetSessionTensor`
* `raw_ops.Greater`
* `raw_ops.GreaterEqual`
* `raw_ops.HSVToRGB`
* `raw_ops.HashTable`
* `raw_ops.HashTableV2`
* `raw_ops.HistogramSummary`
* `raw_ops.Identity`
* `raw_ops.IdentityN`
* `raw_ops.IFFT`
* `raw_ops.IFFT2D`
* `raw_ops.IFFT3D`
* `raw_ops.Imag`
* `raw_ops.ImageProjectiveTransformV2`
* `raw_ops.ImageProjectiveTransformV3`
* `raw_ops.ImmutableConst`
* `raw_ops.InplaceAdd`
* `raw_ops.InplaceSub`
* `raw_ops.InplaceUpdate`
* `raw_ops.InTopK`
* `raw_ops.InTopKV2`
* `raw_ops.InitializeTable`
* `raw_ops.InitializeTableFromDataset`
* `raw_ops.InitializeTableFromTextFile`
* `raw_ops.InitializeTableFromTextFileV2`
* `raw_ops.InitializeTableV2`
* `raw_ops.Inv`
* `raw_ops.Invert`
* `raw_ops.InvertPermutation`
* `raw_ops.InvGrad`
* `raw_ops.IRFFT`
* `raw_ops.IRFFT2D`
* `raw_ops.IRFFT3D`
* `raw_ops.IsFinite`
* `raw_ops.IsNan`
* `raw_ops.IsVariableInitialized`
* `raw_ops.LRN`
* `raw_ops.LeakyRelu`
* `raw_ops.LeakyReluGrad`
* `raw_ops.LeftShift`
* `raw_ops.Less`
* `raw_ops.LessEqual`
* `raw_ops.LinSpace`
* `raw_ops.ListDiff`
* `raw_ops.Log`
* `raw_ops.LogMatrixDeterminant`
* `raw_ops.LogSoftmax`
* `raw_ops.LogicalAnd`
* `raw_ops.LogicalNot`
* `raw_ops.LogicalOr`
* `raw_ops.LookupTableExport`
* `raw_ops.LookupTableExportV2`
* `raw_ops.LookupTableFind`
* `raw_ops.LookupTableFindV2`
* `raw_ops.LookupTableImport`
* `raw_ops.LookupTableImportV2`
* `raw_ops.LookupTableInsert`
* `raw_ops.LookupTableInsertV2`
* `raw_ops.LookupTableRemoveV2`
* `raw_ops.LookupTableSize`
* `raw_ops.LookupTableSizeV2`
* `raw_ops.LoopCond`
* `raw_ops.LRN`
* `raw_ops.MapDataset`
* `raw_ops.MatMul`
* `raw_ops.MatrixBandPart`
* `raw_ops.MatrixDiag`
* `raw_ops.MatrixDiagPart`
* `raw_ops.MatrixDiagPartV2`
* `raw_ops.MatrixDiagPartV3`
* `raw_ops.MatrixDiagV2`
* `raw_ops.MatrixDiagV3`
* `raw_ops.MatrixInverse`
* `raw_ops.MatrixSetDiag`
* `raw_ops.MatrixSetDiagV2`
* `raw_ops.MatrixSetDiagV3`
* `raw_ops.MatrixTriangularSolve`
* `raw_ops.Max`
* `raw_ops.Maximum`
* `raw_ops.MaxPool`
* `raw_ops.MaxPool3D`
* `raw_ops.MaxPool3DGrad`
* `raw_ops.MaxPool3DGradGrad`
* `raw_ops.MaxPoolGrad`
* `raw_ops.MaxPoolGradGrad`
* `raw_ops.MaxPoolGradGradV2`
* `raw_ops.MaxPoolGradV2`
* `raw_ops.MaxPoolGradWithArgmax`
* `raw_ops.MaxPoolV2`
* `raw_ops.MaxPoolWithArgmax`
* `raw_ops.Mean`
* `raw_ops.Merge`
* `raw_ops.MergeSummary`
* `raw_ops.MergeV2Checkpoints`
* `raw_ops.Mfcc`
* `raw_ops.Min`
* `raw_ops.Minimum`
* `raw_ops.MirrorPad`
* `raw_ops.MirrorPadGrad`
* `raw_ops.ModelDataset`
* `raw_ops.Mul`
* `raw_ops.MulNoNan`
* `raw_ops.Multinomial`
* `raw_ops.MutableDenseHashTable`
* `raw_ops.MutableDenseHashTableV2`
* `raw_ops.MutableHashTable`
* `raw_ops.MutableHashTableOfTensors`
* `raw_ops.MutableHashTableOfTensorsV2`
* `raw_ops.MutableHashTableV2`
* `raw_ops.Neg`
* `raw_ops.NextIteration`
* `raw_ops.NonMaxSuppression`
* `raw_ops.NonMaxSuppressionV2`
* `raw_ops.NonMaxSuppressionV3`
* `raw_ops.NonMaxSuppressionV4`
* `raw_ops.NonMaxSuppressionV5`
* `raw_ops.NonMaxSuppressionWithOverlaps`
* `raw_ops.NoOp`
* `raw_ops.NotEqual`
* `raw_ops.OneHot`
* `raw_ops.OnesLike`
* `raw_ops.OptimizeDatasetV2`
* `raw_ops.OptionalFromValue`
* `raw_ops.OptionalGetValue`
* `raw_ops.OptionalHasValue`
* `raw_ops.OptionalNone`
* `raw_ops.Pack`
* `raw_ops.Pad`
* `raw_ops.PadV2`
* `raw_ops.PaddingFIFOQueue`
* `raw_ops.PaddingFIFOQueueV2`
* `raw_ops.PadV2`
* `raw_ops.ParallelConcat`
* `raw_ops.ParallelDynamicStitch`
* `raw_ops.ParseExample`
* `raw_ops.ParseExampleV2`
* `raw_ops.ParseSequenceExample`
* `raw_ops.ParseSequenceExampleV2`
* `raw_ops.ParseSingleExample`
* `raw_ops.ParseSingleSequenceExample`
* `raw_ops.Placeholder`
* `raw_ops.PlaceholderV2`
* `raw_ops.PlaceholderWithDefault`
* `raw_ops.PopulationCount`
* `raw_ops.Pow`
* `raw_ops.PreventGradient`
* `raw_ops.Print`
* `raw_ops.PrintV2`
* `raw_ops.Prod`
* `raw_ops.Qr`
* `raw_ops.QuantizedAdd`
* `raw_ops.QuantizedAvgPool`
* `raw_ops.QuantizedBatchNormWithGlobalNormalization`
* `raw_ops.QuantizedBiasAdd`
* `raw_ops.QuantizedConcat`
* `raw_ops.QuantizedConv2D`
* `raw_ops.QuantizedInstanceNorm`
* `raw_ops.QuantizedMatMul`
* `raw_ops.QuantizedMaxPool`
* `raw_ops.QuantizedMul`
* `raw_ops.QuantizeDownAndShrinkRange`
* `raw_ops.QuantizedRelu`
* `raw_ops.QuantizedRelu6`
* `raw_ops.QuantizedReshape`
* `raw_ops.QuantizedResizeBilinear`
* `raw_ops.QuantizeV2`
* `raw_ops.QueueClose`
* `raw_ops.QueueCloseV2`
* `raw_ops.QueueDequeue`
* `raw_ops.QueueDequeueMany`
* `raw_ops.QueueDequeueManyV2`
* `raw_ops.QueueDequeueUpTo`
* `raw_ops.QueueDequeueUpToV2`
* `raw_ops.QueueDequeueV2`
* `raw_ops.QueueEnqueue`
* `raw_ops.QueueEnqueueMany`
* `raw_ops.QueueEnqueueManyV2`
* `raw_ops.QueueEnqueueV2`
* `raw_ops.QueueIsClosed`
* `raw_ops.QueueIsClosedV2`
* `raw_ops.QueueSize`
* `raw_ops.QueueSizeV2`
* `raw_ops.RFFT`
* `raw_ops.RFFT2D`
* `raw_ops.RFFT3D`
* `raw_ops.RGBToHSV`
* `raw_ops.RaggedBincount`
* `raw_ops.RaggedGather`
* `raw_ops.RaggedRange`
* `raw_ops.RaggedTensorFromVariant`
* `raw_ops.RaggedTensorToSparse`
* `raw_ops.RaggedTensorToTensor`
* `raw_ops.RaggedTensorToVariant`
* `raw_ops.RaggedTensorToVariantGradient`
* `raw_ops.RandomGamma`
* `raw_ops.RandomPoisson`
* `raw_ops.RandomPoissonV2`
* `raw_ops.RandomShuffle`
* `raw_ops.RandomStandardNormal`
* `raw_ops.RandomUniform`
* `raw_ops.RandomUniformInt`
* `raw_ops.Range`
* `raw_ops.Rank`
* `raw_ops.ReadFile`
* `raw_ops.ReadVariableOp`
* `raw_ops.Real`
* `raw_ops.RealDiv`
* `raw_ops.Reciprocal`
* `raw_ops.ReciprocalGrad`
* `raw_ops.Recv`
* `raw_ops.ReduceDataset`
* `raw_ops.ReduceJoin`
* `raw_ops.RefEnter`
* `raw_ops.RefExit`
* `raw_ops.RefIdentity`
* `raw_ops.RefMerge`
* `raw_ops.RefNextIteration`
* `raw_ops.RefSelect`
* `raw_ops.RefSwitch`
* `raw_ops.RegexFullMatch`
* `raw_ops.RegexReplace`
* `raw_ops.Relu`
* `raw_ops.Relu6`
* `raw_ops.Relu6Grad`
* `raw_ops.ReluGrad`
* `raw_ops.RemoteCall`
* `raw_ops.RepeatDataset`
* `raw_ops.RequantizationRange`
* `raw_ops.Requantize`
* `raw_ops.Reshape`
* `raw_ops.ResizeBicubic`
* `raw_ops.ResizeBicubicGrad`
* `raw_ops.ResizeBilinear`
* `raw_ops.ResizeBilinearGrad`
* `raw_ops.ResizeNearestNeighbor`
* `raw_ops.ResizeNearestNeighborGrad`
* `raw_ops.ResourceApplyAdadelta`
* `raw_ops.ResourceApplyAdagrad`
* `raw_ops.ResourceApplyAdagradDA`
* `raw_ops.ResourceApplyAdagradV2`
* `raw_ops.ResourceApplyAdam`
* `raw_ops.ResourceApplyAdaMax`
* `raw_ops.ResourceApplyAdamWithAmsgrad`
* `raw_ops.ResourceApplyAddSign`
* `raw_ops.ResourceApplyCenteredRMSProp`
* `raw_ops.ResourceApplyFtrl`
* `raw_ops.ResourceApplyFtrlV2`
* `raw_ops.ResourceApplyGradientDescent`
* `raw_ops.ResourceApplyKerasMomentum`
* `raw_ops.ResourceApplyMomentum`
* `raw_ops.ResourceApplyPowerSign`
* `raw_ops.ResourceApplyProximalAdagrad`
* `raw_ops.ResourceApplyProximalGradientDescent`
* `raw_ops.ResourceApplyRMSProp`
* `raw_ops.ResourceGather`
* `raw_ops.ResourceGatherNd`
* `raw_ops.ResourceScatterAdd`
* `raw_ops.ResourceScatterDiv`
* `raw_ops.ResourceScatterMax`
* `raw_ops.ResourceScatterMin`
* `raw_ops.ResourceScatterMul`
* `raw_ops.ResourceScatterNdAdd`
* `raw_ops.ResourceScatterNdMax`
* `raw_ops.ResourceScatterNdMin`
* `raw_ops.ResourceScatterNdSub`
* `raw_ops.ResourceScatterNdUpdate`
* `raw_ops.ResourceScatterSub`
* `raw_ops.ResourceScatterUpdate`
* `raw_ops.ResourceSparseApplyAdadelta`
* `raw_ops.ResourceSparseApplyAdagrad`
* `raw_ops.ResourceSparseApplyAdagradDA`
* `raw_ops.ResourceSparseApplyAdagradV2`
* `raw_ops.ResourceSparseApplyCenteredRMSProp`
* `raw_ops.ResourceSparseApplyFtrl`
* `raw_ops.ResourceSparseApplyFtrlV2`
* `raw_ops.ResourceSparseApplyKerasMomentum`
* `raw_ops.ResourceSparseApplyMomentum`
* `raw_ops.ResourceSparseApplyProximalAdagrad`
* `raw_ops.ResourceSparseApplyProximalGradientDescent`
* `raw_ops.ResourceSparseApplyRMSProp`
* `raw_ops.ResourceStridedSliceAssign`
* `raw_ops.Restore`
* `raw_ops.RestoreSlice`
* `raw_ops.RestoreV2`
* `raw_ops.Reverse`
* `raw_ops.ReverseSequence`
* `raw_ops.ReverseV2`
* `raw_ops.RightShift`
* `raw_ops.Roll`
* `raw_ops.Round`
* `raw_ops.Rsqrt`
* `raw_ops.RsqrtGrad`
* `raw_ops.SampleDistortedBoundingBox`
* `raw_ops.SampleDistortedBoundingBoxV2`
* `raw_ops.Save`
* `raw_ops.SaveSlices`
* `raw_ops.SaveV2`
* `raw_ops.ScalarSummary`
* `raw_ops.ScatterNd`
* `raw_ops.ScatterNdAdd`
* `raw_ops.ScatterNdMax`
* `raw_ops.ScatterNdMin`
* `raw_ops.ScatterNdNonAliasingAdd`
* `raw_ops.ScatterNdSub`
* `raw_ops.ScatterNdUpdate`
* `raw_ops.SegmentMax`
* `raw_ops.SegmentMean`
* `raw_ops.SegmentMin`
* `raw_ops.SegmentProd`
* `raw_ops.SegmentSum`
* `raw_ops.Select`
* `raw_ops.SelectV2`
* `raw_ops.Selu`
* `raw_ops.SeluGrad`
* `raw_ops.Send`
* `raw_ops.SerializeTensor`
* `raw_ops.Shape`
* `raw_ops.ShapeN`
* `raw_ops.ShardedFilename`
* `raw_ops.ShardedFilespec`
* `raw_ops.Sigmoid`
* `raw_ops.SigmoidGrad`
* `raw_ops.Sign`
* `raw_ops.Sin`
* `raw_ops.Sinh`
* `raw_ops.Size`
* `raw_ops.Slice`
* `raw_ops.Softmax`
* `raw_ops.SoftmaxCrossEntropyWithLogits`
* `raw_ops.Softplus`
* `raw_ops.SoftplusGrad`
* `raw_ops.Softsign`
* `raw_ops.SoftsignGrad`
* `raw_ops.SpaceToBatch`
* `raw_ops.SpaceToBatchND`
* `raw_ops.SpaceToDepth`
* `raw_ops.SparseAdd`
* `raw_ops.SparseApplyAdadelta`
* `raw_ops.SparseApplyAdagrad`
* `raw_ops.SparseApplyAdagradDA`
* `raw_ops.SparseApplyAdagradV2`
* `raw_ops.SparseApplyCenteredRMSProp`
* `raw_ops.SparseApplyFtrl`
* `raw_ops.SparseApplyFtrlV2`
* `raw_ops.SparseApplyMomentum`
* `raw_ops.SparseApplyProximalAdagrad`
* `raw_ops.SparseApplyProximalGradientDescent`
* `raw_ops.SparseApplyRMSProp`
* `raw_ops.SparseBincount`
* `raw_ops.SparseCross`
* `raw_ops.SparseCrossHashed`
* `raw_ops.SparseCrossV2`
* `raw_ops.SparseFillEmptyRows`
* `raw_ops.SparseFillEmptyRowsGrad`
* `raw_ops.SparseReduceSum`
* `raw_ops.SparseReshape`
* `raw_ops.SparseReorder`
* `raw_ops.SparseSegmentMean`
* `raw_ops.SparseSegmentMeanGrad`
* `raw_ops.SparseSegmentMeanWithNumSegments`
* `raw_ops.SparseSegmentSqrtN`
* `raw_ops.SparseSegmentSqrtNGrad`
* `raw_ops.SparseSegmentSqrtNWithNumSegments`
* `raw_ops.SparseSegmentSum`
* `raw_ops.SparseSegmentSumGrad`
* `raw_ops.SparseSegmentSumWithNumSegments`
* `raw_ops.SparseSlice`
* `raw_ops.SparseSoftmaxCrossEntropyWithLogits`
* `raw_ops.SparseTensorDenseMatMul`
* `raw_ops.SparseToDense`
* `raw_ops.SparseToSparseSetOperation`
* `raw_ops.Split`
* `raw_ops.SplitV`
* `raw_ops.Sqrt`
* `raw_ops.SqrtGrad`
* `raw_ops.Square`
* `raw_ops.SquaredDifference`
* `raw_ops.Squeeze`
* `raw_ops.Stack`
* `raw_ops.StackClose`
* `raw_ops.StackCloseV2`
* `raw_ops.StackPop`
* `raw_ops.StackPopV2`
* `raw_ops.StackPush`
* `raw_ops.StackPushV2`
* `raw_ops.StackV2`
* `raw_ops.StatelessMultinomial`
* `raw_ops.StatelessRandomGammaV2`
* `raw_ops.StatelessRandomGammaV3`
* `raw_ops.StatelessRandomGetAlg`
* `raw_ops.StatelessRandomGetKeyCounter`
* `raw_ops.StatelessRandomGetKeyCounterAlg`
* `raw_ops.StatelessRandomNormal`
* `raw_ops.StatelessRandomNormalV2`
* `raw_ops.StatelessRandomPoisson`
* `raw_ops.StatelessRandomUniform`
* `raw_ops.StatelessRandomUniformFullInt`
* `raw_ops.StatelessRandomUniformFullIntV2`
* `raw_ops.StatelessRandomUniformInt`
* `raw_ops.StatelessRandomUniformIntV2`
* `raw_ops.StatelessRandomUniformV2`
* `raw_ops.StatelessSampleDistortedBoundingBox`
* `raw_ops.StatelessTruncatedNormal`
* `raw_ops.StatelessTruncatedNormalV2`
* `raw_ops.StaticRegexFullMatch`
* `raw_ops.StaticRegexReplace`
* `raw_ops.StopGradient`
* `raw_ops.StridedSlice`
* `raw_ops.StridedSliceAssign`
* `raw_ops.StridedSliceGrad`
* `raw_ops.StringFormat`
* `raw_ops.StringJoin`
* `raw_ops.StringLength`
* `raw_ops.StringLower`
* `raw_ops.StringSplit`
* `raw_ops.StringSplitV2`
* `raw_ops.StringStrip`
* `raw_ops.StringToHashBucket`
* `raw_ops.StringToHashBucketFast`
* `raw_ops.StringToHashBucketStrong`
* `raw_ops.StringToNumber`
* `raw_ops.Sub`
* `raw_ops.Substr`
* `raw_ops.Sum`
* `raw_ops.Switch`
* `raw_ops.SymbolicGradient`
* `raw_ops.TakeDataset`
* `raw_ops.TakeWhileDataset`
* `raw_ops.Tan`
* `raw_ops.Tanh`
* `raw_ops.TanhGrad`
* `raw_ops.TemporaryVariable`
* `raw_ops.TensorArray`
* `raw_ops.TensorArrayClose`
* `raw_ops.TensorArrayCloseV2`
* `raw_ops.TensorArrayCloseV3`
* `raw_ops.TensorArrayConcat`
* `raw_ops.TensorArrayConcatV2`
* `raw_ops.TensorArrayConcatV3`
* `raw_ops.TensorArrayGather`
* `raw_ops.TensorArrayGatherV2`
* `raw_ops.TensorArrayGatherV3`
* `raw_ops.TensorArrayGrad`
* `raw_ops.TensorArrayGradV2`
* `raw_ops.TensorArrayGradV3`
* `raw_ops.TensorArrayGradWithShape`
* `raw_ops.TensorArrayPack`
* `raw_ops.TensorArrayRead`
* `raw_ops.TensorArrayReadV2`
* `raw_ops.TensorArrayReadV3`
* `raw_ops.TensorArrayScatter`
* `raw_ops.TensorArrayScatterV2`
* `raw_ops.TensorArrayScatterV3`
* `raw_ops.TensorArraySize`
* `raw_ops.TensorArraySizeV2`
* `raw_ops.TensorArraySizeV3`
* `raw_ops.TensorArraySplit`
* `raw_ops.TensorArraySplitV2`
* `raw_ops.TensorArraySplitV3`
* `raw_ops.TensorArrayUnpack`
* `raw_ops.TensorArrayV2`
* `raw_ops.TensorArrayV3`
* `raw_ops.TensorArrayWrite`
* `raw_ops.TensorArrayWriteV2`
* `raw_ops.TensorArrayWriteV3`
* `raw_ops.TensorListConcat`
* `raw_ops.TensorListConcatLists`
* `raw_ops.TensorListConcatV2`
* `raw_ops.TensorListElementShape`
* `raw_ops.TensorListFromTensor`
* `raw_ops.TensorListGather`
* `raw_ops.TensorListGetItem`
* `raw_ops.TensorListLength`
* `raw_ops.TensorListPopBack`
* `raw_ops.TensorListPushBack`
* `raw_ops.TensorListPushBackBatch`
* `raw_ops.TensorListReserve`
* `raw_ops.TensorListResize`
* `raw_ops.TensorListScatter`
* `raw_ops.TensorListScatterIntoExistingList`
* `raw_ops.TensorListScatterV2`
* `raw_ops.TensorListSetItem`
* `raw_ops.TensorListSplit`
* `raw_ops.TensorListStack`
* `raw_ops.TensorMapErase`
* `raw_ops.TensorMapHasKey`
* `raw_ops.TensorMapInsert`
* `raw_ops.TensorMapLookup`
* `raw_ops.TensorMapSize`
* `raw_ops.TensorMapStackKeys`
* `raw_ops.TensorScatterAdd`
* `raw_ops.TensorScatterMax`
* `raw_ops.TensorScatterMin`
* `raw_ops.TensorScatterSub`
* `raw_ops.TensorScatterUpdate`
* `raw_ops.TensorSliceDataset`
* `raw_ops.TensorStridedSliceUpdate`
* `raw_ops.Tile`
* `raw_ops.TileGrad`
* `raw_ops.Timestamp`
* `raw_ops.TokenizerFromLogits`
* `raw_ops.TopK`
* `raw_ops.TopKV2`
* `raw_ops.Transpose`
* `raw_ops.TruncateDiv`
* `raw_ops.TruncatedNormal`
* `raw_ops.UnicodeDecode`
* `raw_ops.UnicodeDecodeWithOffsets`
* `raw_ops.UnicodeEncode`
* `raw_ops.UnicodeTranscode`
* `raw_ops.Unique`
* `raw_ops.UniqueV2`
* `raw_ops.UniqueWithCounts`
* `raw_ops.UniqueWithCountsV2`
* `raw_ops.Unpack`
* `raw_ops.UnsortedSegmentJoin`
* `raw_ops.UnsortedSegmentMax`
* `raw_ops.UnsortedSegmentMin`
* `raw_ops.UnsortedSegmentProd`
* `raw_ops.UnsortedSegmentSum`
* `raw_ops.UnwrapDatasetVariant`
* `raw_ops.UpperBound`
* `raw_ops.VarHandleOp`
* `raw_ops.Variable`
* `raw_ops.VariableShape`
* `raw_ops.VariableV2`
* `raw_ops.VarIsInitializedOp`
* `raw_ops.Where`
* `raw_ops.WrapDatasetVariant`
* `raw_ops.WriteFile`
* `raw_ops.Xdivy`
* `raw_ops.Xlog1py`
* `raw_ops.Xlogy`
* `raw_ops.ZerosLike`
## TensorFlow Text and SentencePiece operators
The following
[TensorFlow Text](https://www.tensorflow.org/tutorials/tensorflow_text/intro)
and [SentencePiece](https://github.com/google/sentencepiece) operators are
supported if you use the Python API for conversion and import those libraries.
TF.Text operators:
* `CaseFoldUTF8`
* `ConstrainedSequence`
* `MaxSpanningTree`
* `NormalizeUTF8`
* `NormalizeUTF8WithOffsetsMap`
* `RegexSplitWithOffsets`
* `RougeL`
* `SentenceFragments`
* `SentencepieceOp`
* `SentencepieceTokenizeOp`
* `SentencepieceTokenizeWithOffsetsOp`
* `SentencepieceDetokenizeOp`
* `SentencepieceVocabSizeOp`
* `SplitMergeTokenizeWithOffsets`
* `UnicodeScriptTokenizeWithOffsets`
* `WhitespaceTokenizeWithOffsets`
* `WordpieceTokenizeWithOffsets`
SentencePiece operators:
* `SentencepieceGetPieceSize`
* `SentencepiecePieceToId`
* `SentencepieceIdToPiece`
* `SentencepieceEncodeDense`
* `SentencepieceEncodeSparse`
* `SentencepieceDecode`
The following snippet shows how to convert models with the above operators:
```python
import tensorflow as tf
# These imports are required to load operators' definition.
import tensorflow_text as tf_text
import sentencepiece as spm
converter = tf.lite.TFLiteConverter.from_keras_model(your_model)
converter.target_spec.supported_ops = [
tf.lite.OpsSet.TFLITE_BUILTINS, tf.lite.OpsSet.SELECT_TF_OPS
]
model_data = converter.convert()
```
On the runtime side, it is also required to link the TensorFlow Text or
SentencePiece library into the final app or binary.
## User's defined Operators
*Note: This feature is only available from TensorFlow 2.5 version*
If you
[created your own TensorFlow operators](https://www.tensorflow.org/guide/create_op),
you can also convert models containing them to TensorFlow Lite by listing
required operators in the `experimental_select_user_tf_ops` as following:
```python
import tensorflow as tf
ops_module = tf.load_op_library('./your_ops_library.so')
converter = tf.lite.TFLiteConverter.from_saved_model(your_model)
converter.target_spec.supported_ops = [
tf.lite.OpsSet.TFLITE_BUILTINS, tf.lite.OpsSet.SELECT_TF_OPS
]
converter.target_spec.experimental_select_user_tf_ops = [
'your_op_name1',
'your_op_name2'
]
model_data = converter.convert()
```
On the runtime side, it is also required to link your operators library into the
final app or binary.
## Add TensorFlow core operators to the allowed list.
If you hit the case where the TensorFlow core operators are not in the above
allowed
[list](https://www.tensorflow.org/lite/guide/op_select_allowlist#tensorflow_core_operators),
you can report the feature request at
[here](https://github.com/tensorflow/tensorflow/issues) with the names of the
TensorFlow core operators, not listed in the allowed list.
You can also create own your pull request from the source code. For example, if
you want to add the `raw_ops.StringToNumber` op in the allowed list, there are
three places to update like this
[commit](https://github.com/tensorflow/tensorflow/commit/02e691329517eb5e76522ed8d8bef79ceb082ff8).
(1) Add the operator kernel source code to the `portable_extended_ops_group2`
BUILD rule.
```
filegroup(
name = "portable_extended_ops_group2",
srcs = [
...
+ "string_to_number_op.cc",
...
],
)
```
In order to find the relevant operator kernel source file under the
`tensorflow/core/kernels` directory, you can search the source code location,
which contains the following kernel declaration with the operator name:
```
REGISTER_KERNEL_BUILDER(Name("StringToNumber") \
.Device(DEVICE_CPU) \
.TypeConstraint<type>("out_type"), \
StringToNumberOp<type>)
```
If there are any header files under the `tensorflow/core/kernels` directory,
required in the operator kernel source code, you need to add the header file
into the `portable_extended_ops_headers` BUILD rule as the follows:
```
filegroup(
name = "portable_extended_ops_headers",
srcs = [
...
+ "string_util.h",
...
],
)
```
(2) Add the operator name to the allowed list.
The allowed list is defined in the
`tensorflow/compiler/mlir/lite/delegates/flex/allowlisted_flex_ops.cc`. The
TensorFlow core operator name is need to be listed in order to be allowed
through the Select TF option.
```
static const std::set<std::string>* allowlisted_flex_ops =
new std::set<std::string>({
...
+ "StringToNumber",
...
});
```
Since the above list is sorted in alphabetical order, it makes sure to place the
name in the right place.
(3) Add the operator name to this guide page.
To show the operator inclusion to the other developers, this guide page should
be updated as well. This page is located at the
`tensorflow/lite/g3doc/guide/op_select_allowlist.md`.
```
## TensorFlow core operators
The following is an exhaustive list of TensorFlow core operations that are
supported by TensorFlow Lite runtime with the Select TensorFlow Ops feature.
...
+* `raw_ops.StringToNumber`
...
```
Since the above list is sorted in alphabetical order, it makes sure to place the
name in the right place.
tensorflow/lite/g3doc/guide/ops_compatibility.md
+143
View File
@@ -0,0 +1,143 @@
# TensorFlow Lite and TensorFlow operator compatibility
The machine learning (ML) operators you use in your model can impact the process
of converting a TensorFlow model to TensorFlow Lite format. The TensorFlow Lite
converter supports a limited number of TensorFlow operations used in common
inference models, which means that not every model is directly convertible. The
converter tool allows you to include additional operators, but converting a
model this way also requires you to modify the TensorFlow Lite runtime
environment you use to execute your model, which can limit your ability use
standard runtime deployment options, such as
[Google Play services](../android/play_services.md).
The TensorFlow Lite Converter is designed to analyze model structure and apply
optimizations in order to make it compatible with the directly supported
operators. For example, depending on the ML operators in your model, the
converter may [elide or fuse](../models/convert/operation_fusion.md) those
operators in order to map them to their TensorFlow Lite counterparts.
Even for supported operations, specific usage patterns are sometimes expected,
for performance reasons. The best way to understand how to build a TensorFlow
model that can be used with
TensorFlow Lite is to carefully consider how operations are converted and
optimized, along with the limitations imposed by this process.
## Supported operators
TensorFlow Lite built-in operators are a subset of the operators
that are part of the TensorFlow core library. Your TensorFlow model may
also include custom operators in the form of composite operators
or new operators defined by you. The diagram below shows the relationships
between these operators.
![TensorFlow operators](../images/convert/tf_operators_relationships.png)
From this range of ML model operators, there are 3 types of
models supported by the conversion process:
1. Models with only TensorFlow Lite built-in operator. (**Recommended**)
1. Models with the built-in operators and select TensorFlow
core operators.
1. Models with the built-in operators, TensorFlow core operators and/or
custom operators.
If your model only contains operations that are natively supported by TensorFlow
Lite, you do not need any additional flags to convert it. This is the
recommended path because this type of model will convert smoothly and is simpler
to optimize and run using the default TensorFlow Lite runtime. You also have
more deployment options for your model such as
[Google Play services](../android/play_services.md). You can get started with
the [TensorFlow Lite converter guide](../models/convert/convert_models.md). See
the [TensorFlow Lite Ops page](https://www.tensorflow.org/mlir/tfl_ops) for a
list of built-in operators.
If you need to include select TensorFlow operations from the core library,
you must specify that at conversion and ensure your runtime includes those
operations. See the [Select TensorFlow operators](ops_select.md) topic for
detailed steps.
Whenever possible, avoid the last option of including custom operators in your
converted model. [Custom operators](https://www.tensorflow.org/guide/create_op)
are either operators created by combining multiple primitive TensorFlow core
operators or defining a completely new one. When custom operators are converted,
they can increase the size of the overall model by incurring dependencies
outside of the built-in TensorFlow Lite library. Custom ops, if not specifically
created for mobile or device deployment, can result in worse performance when
deployed to resource constrained devices compared to a server environment.
Finally, just like including select TensorFlow core operators, custom operators
requires you to
[modify the model runtime environment](ops_custom.md#create-and-register-the-operator)
which limits you from taking advantage of standard runtime services such as the
[Google Play services](../android/play_services.md).
## Supported types
Most TensorFlow Lite operations target both floating-point (`float32`) and
quantized (`uint8`, `int8`) inference, but many ops do not yet for other types
like `tf.float16` and strings.
Apart from using different version of the operations, the other difference
between floating-point and quantized models is the way they are converted.
Quantized conversion requires dynamic range information for tensors. This
requires "fake-quantization" during model training, getting range information
via a calibration data set, or doing "on-the-fly" range estimation. See
[quantization](../performance/model_optimization.md) for more details.
## Straight-forward conversions, constant-folding and fusing
A number of TensorFlow operations can be processed by TensorFlow Lite even
though they have no direct equivalent. This is the case for operations that can
be simply removed from the graph (`tf.identity`), replaced by tensors
(`tf.placeholder`), or fused into more complex operations (`tf.nn.bias_add`).
Even some supported operations may sometimes be removed through one of these
processes.
Here is a non-exhaustive list of TensorFlow operations that are usually removed
from the graph:
* `tf.add`
* `tf.debugging.check_numerics`
* `tf.constant`
* `tf.div`
* `tf.divide`
* `tf.fake_quant_with_min_max_args`
* `tf.fake_quant_with_min_max_vars`
* `tf.identity`
* `tf.maximum`
* `tf.minimum`
* `tf.multiply`
* `tf.no_op`
* `tf.placeholder`
* `tf.placeholder_with_default`
* `tf.realdiv`
* `tf.reduce_max`
* `tf.reduce_min`
* `tf.reduce_sum`
* `tf.rsqrt`
* `tf.shape`
* `tf.sqrt`
* `tf.square`
* `tf.subtract`
* `tf.tile`
* `tf.nn.batch_norm_with_global_normalization`
* `tf.nn.bias_add`
* `tf.nn.fused_batch_norm`
* `tf.nn.relu`
* `tf.nn.relu6`
Note: Many of those operations don't have TensorFlow Lite equivalents, and the
corresponding model will not be convertible if they can't be elided or fused.
## Experimental Operations
The following TensorFlow Lite operations are present, but not ready for custom
models:
* `CALL`
* `CONCAT_EMBEDDINGS`
* `CUSTOM`
* `EMBEDDING_LOOKUP_SPARSE`
* `HASHTABLE_LOOKUP`
* `LSH_PROJECTION`
* `SKIP_GRAM`
* `SVDF`
tensorflow/lite/g3doc/guide/ops_custom.md
+650
View File
@@ -0,0 +1,650 @@
# Custom operators
Since the TensorFlow Lite builtin operator library only supports a limited
number of TensorFlow operators, not every model is convertible. For details,
refer to [operator compatibility](ops_compatibility.md).
To allow conversion, users can provide their own custom implementation of an
unsupported TensorFlow operator in TensorFlow Lite, known as a custom operator.
*If instead, you wish to combine a series of unsupported (or supported)
TensorFlow operators into a single fused optimized custom operator, refer to
[operator fusing](https://www.tensorflow.org/lite/models/convert/operation_fusion).*
Using custom operators consists of four steps.
* [Create a TensorFlow Model.](#create-a-tensorflow-model) Make sure the Saved
Model (or Graph Def) refers to the correctly named TensorFlow Lite operator.
* [Convert to a TensorFlow Lite Model.](#convert-to-a-tensorflow-lite-model)
Make sure you set the right TensorFlow Lite converter attribute in order to
successfully convert the model.
* [Create and register the operator.](#create-and-register-the-operator) This
is so that the TensorFlow Lite runtime knows how to map your operator and
parameters in your graph to executable C/C++ code.
* [Test and profile your operator.](#test-and-profile-your-operator) If you
wish to test just your custom operator, it is best to create a model with
just your custom operator and use the
[benchmark_model](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/tools/benchmark/benchmark_model.cc)
program.
Let’s walk through an end-to-end example of running a model with a custom
operator `tf.atan` (named as `Atan`, refer to #create-a-tensorflow-model) which
is supported in TensorFlow, but unsupported in TensorFlow Lite.
Note: The `tf.atan` function is **not** a custom operator. It is a regular
operator
which is supported by both TensorFlow and TensorFlow Lite. But we **assume**
that it is a custom operator in the following example in order to demonstrate a
simple workflow.
The TensorFlow Text operator is an example of a custom operator. See the
<a href="https://tensorflow.org/text/guide/text_tf_lite" class="external">
Convert TF Text to TF Lite</a> tutorial for a code example.
## Example: Custom `Atan` operator
Let’s walk through an example of supporting a TensorFlow operator that
TensorFlow Lite does not have. Assume we are using the `Atan` operator and that
we are building a very simple model for a function `y = atan(x + offset)`, where
`offset` is trainable.
### Create a TensorFlow Model
The following code snippet trains a simple TensorFlow model. This model just
contains a custom operator named `Atan`, which is a function `y = atan(x +
offset)`, where `offset` is trainable.
```python
import tensorflow as tf
# Define training dataset and variables
x = [-8, 0.5, 2, 2.2, 201]
y = [-1.4288993, 0.98279375, 1.2490457, 1.2679114, 1.5658458]
offset = tf.Variable(0.0)
# Define a simple model which just contains a custom operator named `Atan`
@tf.function(input_signature=[tf.TensorSpec.from_tensor(tf.constant(x))])
def atan(x):
return tf.atan(x + offset, name="Atan")
# Train model
optimizer = tf.optimizers.Adam(0.01)
def train(x, y):
with tf.GradientTape() as t:
predicted_y = atan(x)
loss = tf.reduce_sum(tf.square(predicted_y - y))
grads = t.gradient(loss, [offset])
optimizer.apply_gradients(zip(grads, [offset]))
for i in range(1000):
train(x, y)
print("The actual offset is: 1.0")
print("The predicted offset is:", offset.numpy())
```
```python
The actual offset is: 1.0
The predicted offset is: 0.99999905
```
At this point, if you try to generate a TensorFlow Lite model with the default
converter flags, you will get the following error message:
```none
Error:
error: 'tf.Atan' op is neither a custom op nor a flex op.
```
### Convert to a TensorFlow Lite Model
Create a TensorFlow Lite model with custom operators, by setting the converter
attribute `allow_custom_ops` as shown below:
<pre>
converter = tf.lite.TFLiteConverter.from_concrete_functions([atan.get_concrete_function()], atan)
<b>converter.allow_custom_ops = True</b>
tflite_model = converter.convert()
</pre>
At this point, if you run it with the default interpreter using commands such as
follows:
```python
interpreter = tf.lite.Interpreter(model_content=tflite_model)
interpreter.allocate_tensors()
```
You will still get the error:
```none
Encountered unresolved custom op: Atan.
```
### Create and register the operator.
```c++
#include "tensorflow/lite/c/c_api.h"
#include "tensorflow/lite/c/c_api_opaque.h"
```
TensorFlow Lite custom operators are defined using a simple pure-C API that
consists of an opaque type (`TfLiteOperator`) and related functions.
`TfLiteOperator` is an opaque type:
```c++
typedef struct TfLiteOperator TfLiteOperator;
```
`TfLiteOperator` stores the operator's identity and implementation.
(Note that the operator is distinct from its operands, which are stored in the
TF Lite graph nodes for nodes that call the operator.)
Instances of this type are constructed with calls to
`TfLiteOperatorCreate` and can be destroyed by calling
`TfLiteOperatorDelete`.
The operator's identity is set via the parameters to the constructor function
`TfLiteOperatorCreate`:
```c++
TfLiteOperator*
TfLiteOperatorCreate(
TfLiteBuiltinOperator builtin_code, // Normally `TfLiteBuiltinCustom`.
const char* custom_name, // The name of the custom op.
int version // Normally `1` for the first version of a custom op.
);
```
The operator implementation can define "methods" with the following signatures.
All of these methods are optional, but for an operator to be successfully
evaluated, the operator implementation needs to define and set (using the setter
functions) at least the `Prepare` and `Invoke` methods.
```c++
// Initializes the op from serialized data.
void* Init(TfLiteOpaqueContext* context, const char* buffer, size_t length);
// Deallocates the op.
// The pointer `buffer` is the data previously returned by an Init invocation.
void Free(TfLiteOpaqueContext* context, void* buffer);
// Called when the inputs that this node depends on have been resized.
TfLiteStatus Prepare(TfLiteOpaqueContext* context, TfLiteOpaqueNode* node);
// Called when the node is executed. (Should read node inputs and write to
// node outputs).
TfLiteStatus Invoke(TfLiteOpaqueContext* context, TfLiteOpaqueNode* node);
// Retrieves the async kernel.
TfLiteAsyncKernel AsyncKernel(TfLiteOpaqueContext* context,
TfLiteOpaqueNode* node);
```
The function *names* (or namespace prefixes, for C++) in your op implementation
don't have to match the function names in the above code snippet, since the TF
Lite custom ops API will only use their addresses. Indeed we recommend that you
declare them in an anonymous namespace or as static functions.
But it is a good idea to include your operator name as a namespace or prefix on
these function names:
<div>
<devsite-selector>
<section>
<h3>C++</h3>
<p><pre class="prettyprint lang-cpp">
namespace my_namespace::my_custom_op {
void* Init(TfLiteOpaqueContext* context,
const char* buffer, size_t length) { ... }
// ... plus definitions of Free, Prepare, and Invoke ...
}
</pre></p>
</section>
<section>
<h3>C</h3>
<p><pre class="prettyprint lang-cpp">
void* MyCustomOpInit(TfLiteOpaqueContext* context,
const char* buffer, size_t length) { ... }
// ... plus definitions of MyCustomOpFree, MyCustomOpPrepare, and
// MyCustomOpInvoke.
</pre></p>
</section>
</devsite-selector>
</div>
Since this is a C API, these "methods" are implemented as C function pointers in
the `TfLiteOperator` type, which are set by passing the addresses of
your implementation functions to the corresponding setter functions
`TfLiteOperatorSet`*MethodName*:
```c++
void TfLiteOperatorSetInit(
TfLiteOperator* registration,
void* (*init)(TfLiteOpaqueContext* context, const char* buffer,
size_t length));
void TfLiteOperatorSetFree(
TfLiteOperator* registration,
void (*free)(TfLiteOpaqueContext* context, void* data));
void TfLiteOperatorSetPrepare(
TfLiteOperator* registration,
TfLiteStatus (*prepare)(TfLiteOpaqueContext* context,
TfLiteOpaqueNode* node));
void TfLiteOperatorSetInvoke(
TfLiteOperator* registration,
TfLiteStatus (*invoke)(TfLiteOpaqueContext* context,
TfLiteOpaqueNode* node));
void TfLiteOperatorSetAsyncKernel(
TfLiteOperator* registration,
struct TfLiteAsyncKernel* (*async_kernel)(TfLiteOpaqueContext* context,
TfLiteOpaqueNode* node));
```
Refer to
[`common.h`](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/core/c/common.h)
for details on `TfLiteContext` and `TfLiteNode`. `TfLiteContext` provides error
reporting facilities and access to global objects, including all the tensors.
`TfLiteNode` allows operator implementations to access their inputs and outputs.
When the interpreter loads a model, it calls the `Init()` method once for each
node in the graph. A given `Init()` will be called more than once if the op is
used multiple times in the graph. For custom ops a configuration buffer will be
provided, containing a flexbuffer that maps parameter names to their values. The
buffer is empty for builtin ops because the interpreter has already parsed the
op parameters. Kernel implementations that require state should initialize it
here and transfer ownership to the caller. For each `Init()` call, there will be
a corresponding call to `Free()`, allowing implementations to dispose of the
buffer they might have allocated in `Init()`.
Whenever the input tensors are resized, the interpreter will go through the
graph notifying implementations of the change. This gives them the chance to
resize their internal buffer, check validity of input shapes and types, and
recalculate output shapes. This is all done through the `Prepare()` method, and
implementations can access their state using
`TfLiteOpaqueNodeGetUserData(node)`.
Finally, each time inference runs, the interpreter traverses the graph calling
the `Invoke()` method, and here too the state is available as
`TfLiteOpaqueNodeGetUserData(node)`.
Custom ops can be implemented by defining those "method" functions, and then
defining a function that returns an instance of `TfLiteOperator`
constructed by calling `TfLiteOperatorCreate` and then the relevant
setter methods:
<div>
<devsite-selector>
<section>
<h3>C++</h3>
<p><pre class="prettyprint lang-cpp">
namespace my_namespace::my_custom_op {
namespace {
void* Init(TfLiteOpaqueContext* context,
const char* buffer, size_t length) { ... }
void Free(TfLiteOpaqueContext* context, void* buffer) { ... }
TfLiteStatus Prepare(TfLiteOpaqueContext* context,
TfLiteOpaqueNode* node) { ... }
TfLiteStatus Invoke(TfLiteOpaqueContext* context,
TfLiteOpaqueNode* node) {... }
};
const TfLiteOperator* MyCustomOpRegistrationExternal() {
// Singleton instance, intentionally never destroyed.
static const TfLiteOperator* my_custom_op = ()[] {
TfLiteOperator* r =
TfLiteOperatorCreate(
kTfLiteBuiltinCustom, "MyCustomOp", /*version=*/ 1);
TfLiteOperatorSetInit(r, Init);
TfLiteOperatorSetFree(r, Free);
TfLiteOperatorSetPrepare(r, Prepare);
TfLiteOperatorSetInvoke(r, Eval);
return r;
};
return my_custom_op;
}
} // namespace my_namespace
</pre></p>
</section>
<section>
<h3>C</h3>
<p><pre class="prettyprint lang-cpp">
static void* MyCustomOpInit(TfLiteOpaqueContext* context, const char* buffer,
size_t length) { ... }
static void MyCustomOpFree(TfLiteOpaqueContext* context, void* buffer) { ... }
static TfLiteStatus MyCustomOpPrepare(TfLiteOpaqueContext* context,
TfLiteOpaqueNode* node) { ... }
static TfLiteStatus MyCustomOpInvoke(TfLiteOpaqueContext* context,
TfLiteOpaqueNode* node) {... }
static TfLiteOperator* MyCustomOpCreate() {
const TfLiteOperator* r =
TfLiteOperatorCreate(
kTfLiteBuiltinCustom, "MyCustomOp", /*version=*/ 1);
TfLiteOperatorSetInit(r, MyCustomOpInit);
TfLiteOperatorSetFree(r, MyCustomOpFree);
TfLiteOperatorSetPrepare(r, MyCustomOpPrepare);
TfLiteOperatorSetInvoke(r, MyCustomOpEval);
return r;
}
const TfLiteOperator* MyCustomOpRegistrationExternal() {
// Singleton instance, intentionally never destroyed.
static const TfLiteOperator* my_custom_op = MyCustomOpCreate();
return my_custom_op;
}
</pre></p>
</section>
</devsite-selector>
</div>
Note that registration is not automatic and an explicit call to your
`MyCustomOpRegistration` function should be made (see details below). While the
standard `BuiltinOpResolver` (available from the `:builtin_ops` target) takes
care of the registration of builtins, custom ops will have to be collected in
separate custom libraries.
### Defining the kernel in the TensorFlow Lite runtime
All we need to do to use the op in TensorFlow Lite is define two functions
(`Prepare` and `Eval`), and a third to construct a `TfLiteOperator`:
<div>
<devsite-selector>
<section>
<h3>C++</h3>
<p><pre class="prettyprint lang-cpp">
namespace atan_op {
namespace {
TfLiteStatus AtanPrepare(TfLiteOpaqueContext* context, TfLiteOpaqueNode* node) {
TF_LITE_OPAQUE_ENSURE_EQ(context, TfLiteOpaqueNodeNumInputs(node), 1);
TF_LITE_OPAQUE_ENSURE_EQ(context, TfLiteOpaqueNodeNumOutputs(node), 1);
const TfLiteOpaqueTensor* input = TfLiteOpaqueNodeGetInput(context, node, 0);
TfLiteOpaqueTensor* output = TfLiteOpaqueNodeGetOutput(context, node, 0);
int num_dims = TfLiteOpaqueTensorNumDimensions(input);
TfLiteIntArray* output_size = TfLiteIntArrayCreate(num_dims);
for (int i=0; i < num_dims; ++i) {
output_size->data[i] = input->dims->data[i];
}
return TfLiteOpaqueContextResizeTensor(context, output, output_size);
}
TfLiteStatus AtanEval(TfLiteOpaqueContext* context, TfLiteOpaqueNode* node) {
const TfLiteOpaqueTensor* input = TfLiteOpaqueNodeGetInput(context, node, 0);
TfLiteOpaqueTensor* output = TfLiteOpaqueNodeGetOutput(context, node, 0);
float* input_data = static_cast<float*>(TfLiteOpaqueTensorData(input));
float* output_data = static_cast<float*>(TfLiteOpaqueTensorData(output));
size_t count = 1;
int num_dims = TfLiteOpaqueTensorNumDimensions(input);
for (int i = 0; i < num_dims; ++i) {
count *= input->dims->data[i];
}
for (size_t i = 0; i < count; ++i) {
output_data[i] = atan(input_data[i]);
}
return kTfLiteOk;
}
} // anonymous namespace
const TfLiteOperator* AtanOpRegistrationExternal() {
// Singleton instance, intentionally never destroyed.
static const TfLiteOperator* atan_op = ()[] {
auto* r = TfLiteOperatorCreate(
kTfLiteBuiltinCustom, "ATAN", /*version=*/ 1);
TfLiteOperatorSetPrepare(r, Prepare);
TfLiteOperatorSetInvoke(r, Eval);
return r;
};
return atan_op;
}
} // namespace atan_op
</pre></p>
</section>
<section>
<h3>C</h3>
<p><pre class="prettyprint lang-cpp">
static TfLiteStatus AtanPrepare(TfLiteOpaqueContext* context, TfLiteOpaqueNode* node) {
TF_LITE_OPAQUE_ENSURE_EQ(context, TfLiteOpaqueNodeNumInputs(node), 1);
TF_LITE_OPAQUE_ENSURE_EQ(context, TfLiteOpaqueNodeNumOutputs(node), 1);
const TfLiteOpaqueTensor* input = TfLiteOpaqueNodeGetInput(context, node, 0);
TfLiteOpaqueTensor* output = TfLiteOpaqueNodeGetOutput(context, node, 0);
int num_dims = TfLiteOpaqueTensorNumDimensions(input);
TfLiteIntArray* output_size = TfLiteIntArrayCreate(num_dims);
for (int i = 0; i < num_dims; ++i) {
output_size->data[i] = input->dims->data[i];
}
return TfLiteOpaqueContextResizeTensor(context, output, output_size);
}
static TfLiteStatus AtanEval(TfLiteOpaqueContext* context, TfLiteOpaqueNode* node) {
const TfLiteOpaqueTensor* input = TfLiteOpaqueNodeGetInput(context, node, 0);
TfLiteOpaqueTensor* output = TfLiteOpaqueNodeGetOutput(context, node, 0);
float* input_data = static_cast<float*>(TfLiteOpaqueTensorData(input));
float* output_data = static_cast<float*>(TfLiteOpaqueTensorData(output));
size_t count = 1;
int num_dims = TfLiteOpaqueTensorNumDimensions(input);
for (int i = 0; i < num_dims; ++i) {
count *= input->dims->data[i];
}
for (size_t i = 0; i < count; ++i) {
output_data[i] = atan(input_data[i]);
}
return kTfLiteOk;
}
static const TfLiteOperator* AtanOpCreate() {
TfLiteOperator* r = TfLiteOperatorCreate(
kTfLiteBuiltinCustom, "ATAN", /*version=*/ 1);
TfLiteOperatorSetPrepare(r, Prepare);
TfLiteOperatorSetInvoke(r, Eval);
return r;
}
const TfLiteOperator* AtanOpRegistrationExternal() {
// Singleton instance, intentionally never destroyed.
static const TfLiteOperator* atan_op = AtanOpCreate();
return atan_op;
}
</pre></p>
</section>
</devsite-selector>
</div>
When initializing the `OpResolver`, add the custom op into the resolver (see
below for an example). This will register the operator with Tensorflow Lite so
that TensorFlow Lite can use the new implementation. Note that the last two
arguments in `TfLiteRegistration` correspond to the `AtanPrepare` and `AtanEval`
functions you defined for the custom op. If you used `AtanInit` and `AtanFree`
functions to initialize variables used in the op and to free up space,
respectively, then they would be added to the first two arguments of
`TfLiteRegistration`; those arguments are set to `nullptr` in this example.
### Register the operator with the kernel library
Now we need to register the operator with the kernel library. This is done with
an `OpResolver`. Behind the scenes, the interpreter will load a library of
kernels which will be assigned to execute each of the operators in the model.
While the default library only contains builtin kernels, it is possible to
replace/augment it with a custom library op operators.
The `OpResolver` class, which translates operator codes and names into actual
code, is defined like this:
```c++
class OpResolver {
public:
virtual TfLiteRegistration* FindOp(tflite::BuiltinOperator op) const = 0;
virtual TfLiteRegistration* FindOp(const char* op) const = 0;
...
};
```
Note that for backwards compatibility, this class uses the older concrete type
`TfLiteRegistration` rather than the opaque type `TfLiteOperator`,
but the `TfLiteRegistration` struct contains a `registration_external` field of
type `TfLiteOperator*`.
The `MutableOpResolver` and `BuiltinOpResolver` classes are derived from
`OpResolver`:
```c++
class MutableOpResolver : public OpResolver {
public:
MutableOpResolver(); // Constructs an initially empty op resolver.
void AddAll(const MutableOpResolver& other);
...
};
class BuiltinOpResolver : public MutableOpResolver {
public:
BuiltinOpResolver(); // Constructs an op resolver with all the builtin ops.
};
```
Regular usage (without custom ops) requires that you use the `BuiltinOpResolver`
and write:
```c++
tflite::ops::builtin::BuiltinOpResolver resolver;
```
To add the custom op created above, you can instead use a `MutableOpResolver`,
and call `AddCustom` (before you pass the resolver to the
`InterpreterBuilder`):
```c++
tflite::ops::builtin::MutableOpResolver resolver;
resolver.AddAll(tflite::ops::builtin::BuiltinOpResolver());
tflite::AddOp(&resolver, AtanOpRegistration());
```
If the set of builtin ops is deemed to be too large, a new `OpResolver` could be
code-generated based on a given subset of ops, possibly only the ones contained
in a given model. This is the equivalent of TensorFlow's selective registration
(and a simple version of it is available in the `tools` directory).
If you want to define your custom operators in Java, you would currently need to
build your own custom JNI layer and compile your own AAR
[in this jni code](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/java/src/main/native/nativeinterpreterwrapper_jni.cc).
Similarly, if you wish to define these operators available in Python you can
place your registrations in the
[Python wrapper code](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/python/interpreter_wrapper/interpreter_wrapper.cc).
Note that a similar process as above can be followed for supporting a set of
operations instead of a single operator. Just add as many `AddCustom` operators
as you need. In addition, `MutableOpResolver` also allows you to override
implementations of builtins by using `AddBuiltin`.
### Test and profile your operator
To profile your op with the TensorFlow Lite benchmark tool, you can use the
[benchmark model tool](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/tools/benchmark#tflite-model-benchmark-tool)
for TensorFlow Lite. For testing purposes, you can make your local build of
TensorFlow Lite aware of your custom op by adding the appropriate `AddCustom`
call (as show above) to
[register.cc](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/core/kernels/register.cc)
## Best practices
1. Optimize memory allocations and de-allocations cautiously. Allocating memory
in `Prepare` is more efficient than in `Invoke`, and allocating memory
before a loop is better than in every iteration. Use temporary tensors data
rather than mallocing yourself (see item 2). Use pointers/references instead
of copying as much as possible.
2. If a data structure will persist during the entire operation, we advise
pre-allocating the memory using temporary tensors. You may need to use an
OpData struct to reference the tensor indices in other functions. See the
example in the
[kernel for convolution](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/kernels/conv.cc).
A sample code snippet is below.
```c++
struct MyOpData {
int temp_tensor_index;
...
};
void* Init(TfLiteOpaqueContext* context,
const char* buffer, size_t length) {
auto* op_data = new MyOpData{};
...
return op_data;
}
void Free(TfLiteOpaqueContext* context, void* buffer) {
...
delete reinterpret_cast<MyOpData*>(buffer);
}
TfLiteStatus Prepare(TfLiteOpaqueContext* context,
TfLiteOpaqueNode* node) {
...
auto* op_data =
reinterpret_cast<MyOpData*>(TfLiteOpaqueNodeGetUserData(node));
const int num_temporaries = 1;
int temporary_tensor_indices[num_temporaries];
TfLiteOpaqueTensorBuilder* builder = TfLiteOpaqueTensorBuilderCreate();
TfLiteOpaqueTensorBuilderSetType(builder, kTfLiteFloat32);
TfLiteOpaqueTensorBuilderSetAllocationType(builder, kTfLiteArenaRw);
TfLiteOpaqueContextAddTensor(context, builder,
&temporary_tensor_indices[0]);
TfLiteOpaqueTensorBuilderDelete(builder);
TfLiteOpaqueNodeSetTemporaries(node, temporary_tensor_indices,
num_temporaries);
op_data->temp_tensor_index = temporary_tensor_indices[0];
...
return kTfLiteOk;
}
TfLiteStatus Invoke(TfLiteOpaqueContext* context,
TfLiteOpaqueNode* node) {
...
auto* op_data = reinterpret_cast<MyOpData*>(
TfLiteOpaqueNodeGetUserData(node));
TfLiteOpaqueTensor* temp_tensor =
TfLiteOpaqueContextGetOpaqueTensor(context,
op_data->temp_tensor_index);
TF_LITE_OPAQUE_ENSURE(context,
TfLiteTensorType(temp_tensor) == kTfLiteFloat32);
TF_LITE_OPAQUE_ENSURE(context,
TfLiteTensorGetAllocationType(temp_Tensor) == kTfLiteArenaRw);
void *temp_data = TfLiteTensorData(temp_tensor);
TF_LITE_OPAQUE_ENSURE(context, temp_data != nullptr);
...
return kTfLiteOk;
}
```
3. If it doesn't cost too much wasted memory, prefer using a static fixed size
array (or a pre-allocated `std::vector` in `Resize`) rather than using a
dynamically allocated `std::vector` every iteration of execution.
4. Avoid instantiating standard library container templates that don't already
exist, because they affect binary size. For example, if you need a
`std::map` in your operation that doesn't exist in other kernels, using a
`std::vector` with direct indexing mapping could work while keeping the
binary size small. See what other kernels use to gain insight (or ask).
5. Check the pointer to the memory returned by `malloc`. If this pointer is
`nullptr`, no operations should be performed using that pointer. If you
`malloc` in a function and have an error exit, deallocate memory before you
exit.
6. Use `TF_LITE_OPAQUE_ENSURE(context, condition)` to check for a specific
condition. Your code must not leave memory hanging when
`TF_LITE_OPAQUE_ENSURE` is used, i.e., these macros should be used before
any resources are allocated that will leak.
tensorflow/lite/g3doc/guide/ops_select.md
+298
View File
@@ -0,0 +1,298 @@
# Select TensorFlow operators
Since the TensorFlow Lite builtin operator library only supports a limited
number of TensorFlow operators, not every model is convertible. For details,
refer to [operator compatibility](ops_compatibility.md).
To allow conversion, users can enable the usage of
[certain TensorFlow ops](op_select_allowlist.md) in their TensorFlow Lite model.
However, running TensorFlow Lite models with TensorFlow ops requires pulling in
the core TensorFlow runtime, which increases the TensorFlow Lite interpreter
binary size. For Android, you can avoid this by selectively building only
required Tensorflow ops. For the details, refer to
[reduce binary size](../guide/reduce_binary_size.md).
This document outlines how to [convert](#convert_a_model) and
[run](#run_inference) a TensorFlow Lite model containing TensorFlow ops on a
platform of your choice. It also discusses
[performance and size metrics](#metrics) and
[known limitations](#known_limitations).
## Convert a model
The following example shows how to generate a TensorFlow Lite model with select
TensorFlow ops.
```python
import tensorflow as tf
converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
converter.target_spec.supported_ops = [
tf.lite.OpsSet.TFLITE_BUILTINS, # enable TensorFlow Lite ops.
tf.lite.OpsSet.SELECT_TF_OPS # enable TensorFlow ops.
]
tflite_model = converter.convert()
open("converted_model.tflite", "wb").write(tflite_model)
```
## Run Inference
When using a TensorFlow Lite model that has been converted with support for
select TensorFlow ops, the client must also use a TensorFlow Lite runtime that
includes the necessary library of TensorFlow ops.
### Android AAR
To reduce the binary size, please build your own custom AAR files as guided in
the [next section](#building-the-android-aar). If the binary size is not a
considerable concern, we recommend using the prebuilt
[AAR with TensorFlow ops hosted at MavenCentral](https://search.maven.org/artifact/org.tensorflow/tensorflow-lite-select-tf-ops).
You can specify this in your `build.gradle` dependencies by adding it alongside
the standard TensorFlow Lite AAR as follows:
```build
dependencies {
implementation 'org.tensorflow:tensorflow-lite:0.0.0-nightly-SNAPSHOT'
// This dependency adds the necessary TF op support.
implementation 'org.tensorflow:tensorflow-lite-select-tf-ops:0.0.0-nightly-SNAPSHOT'
}
```
To use nightly snapshots, make sure that you have added
[Sonatype snapshot repository](../android/lite_build.md#use_nightly_snapshots).
Once you've added the dependency, the necessary delegate for handling the
graph's TensorFlow ops should be automatically installed for graphs that require
them.
*Note*: The TensorFlow ops dependency is relatively large, so you'll probably
want to filter out unnecessary x86 ABIs in your `.gradle` file by setting up
your `abiFilters`.
```build
android {
defaultConfig {
ndk {
abiFilters 'armeabi-v7a', 'arm64-v8a'
}
}
}
```
#### Building the Android AAR
For reducing the binary size or other advanced cases, you can also build the
library manually. Assuming a
[working TensorFlow Lite build environment](../android/quickstart.md), build the
Android AAR with select TensorFlow ops as follows:
```sh
sh tensorflow/lite/tools/build_aar.sh \
--input_models=/a/b/model_one.tflite,/c/d/model_two.tflite \
--target_archs=x86,x86_64,arm64-v8a,armeabi-v7a
```
This will generate the AAR file `bazel-bin/tmp/tensorflow-lite.aar` for
TensorFlow Lite built-in and custom ops; and generate the AAR file
`bazel-bin/tmp/tensorflow-lite-select-tf-ops.aar` for TensorFlow ops. If you
don't have a working build environment, You can also
[build above files with docker](../guide/reduce_binary_size.md#selectively_build_tensorflow_lite_with_docker).
From there, you can either import the AAR files directly into your project, or
publish the custom AAR files to your local Maven repository:
```sh
mvn install:install-file \
-Dfile=bazel-bin/tmp/tensorflow-lite.aar \
-DgroupId=org.tensorflow \
-DartifactId=tensorflow-lite -Dversion=0.1.100 -Dpackaging=aar
mvn install:install-file \
-Dfile=bazel-bin/tmp/tensorflow-lite-select-tf-ops.aar \
-DgroupId=org.tensorflow \
-DartifactId=tensorflow-lite-select-tf-ops -Dversion=0.1.100 -Dpackaging=aar
```
Finally, in your app's `build.gradle`, ensure you have the `mavenLocal()`
dependency and replace the standard TensorFlow Lite dependency with the one that
has support for select TensorFlow ops:
```build
allprojects {
repositories {
mavenCentral()
maven { // Only for snapshot artifacts
name 'ossrh-snapshot'
url 'https://oss.sonatype.org/content/repositories/snapshots'
}
mavenLocal()
}
}
dependencies {
implementation 'org.tensorflow:tensorflow-lite:0.1.100'
implementation 'org.tensorflow:tensorflow-lite-select-tf-ops:0.1.100'
}
```
### iOS
#### Using CocoaPods
TensorFlow Lite provides nightly prebuilt select TF ops CocoaPods for `arm64`,
which you can depend on alongside the `TensorFlowLiteSwift` or
`TensorFlowLiteObjC` CocoaPods.
*Note*: If you need to use select TF ops in an `x86_64` simulator, you can build
the select ops framework yourself. See [Using Bazel + Xcode](#using_bazel_xcode)
section for more details.
```ruby
# In your Podfile target:
pod 'TensorFlowLiteSwift' # or 'TensorFlowLiteObjC'
pod 'TensorFlowLiteSelectTfOps', '~> 0.0.1-nightly'
```
After running `pod install`, you need to provide an additional linker flag to
force load the select TF ops framework into your project. In your Xcode project,
go to `Build Settings` -> `Other Linker Flags`, and add:
For versions >= 2.9.0:
```text
-force_load $(SRCROOT)/Pods/TensorFlowLiteSelectTfOps/Frameworks/TensorFlowLiteSelectTfOps.xcframework/ios-arm64/TensorFlowLiteSelectTfOps.framework/TensorFlowLiteSelectTfOps
```
For versions < 2.9.0:
```text
-force_load $(SRCROOT)/Pods/TensorFlowLiteSelectTfOps/Frameworks/TensorFlowLiteSelectTfOps.framework/TensorFlowLiteSelectTfOps
```
You should then be able to run any models converted with the `SELECT_TF_OPS` in
your iOS app. For example, you can modify the
[Image Classification iOS app](https://github.com/tensorflow/examples/tree/master/lite/examples/image_classification/ios)
to test the select TF ops feature.
* Replace the model file with the one converted with `SELECT_TF_OPS` enabled.
* Add `TensorFlowLiteSelectTfOps` dependency to the `Podfile` as instructed.
* Add the additional linker flag as above.
* Run the example app and see if the model works correctly.
#### Using Bazel + Xcode
TensorFlow Lite with select TensorFlow ops for iOS can be built using Bazel.
First, follow the [iOS build instructions](build_ios.md) to configure your Bazel
workspace and `.bazelrc` file correctly.
Once you have configured the workspace with iOS support enabled, you can use the
following command to build the select TF ops addon framework, which can be added
on top of the regular `TensorFlowLiteC.framework`. Note that the select TF ops
framework cannot be built for `i386` architecture, so you need to explicitly
provide the list of target architectures excluding `i386`.
```sh
bazel build -c opt --config=ios --ios_multi_cpus=arm64,x86_64 \
//tensorflow/lite/ios:TensorFlowLiteSelectTfOps_framework
```
This will generate the framework under `bazel-bin/tensorflow/lite/ios/`
directory. You can add this new framework to your Xcode project by following
similar steps described under the
[Xcode project settings](./build_ios.md#modify_xcode_project_settings_directly)
section in the iOS build guide.
After adding the framework into your app project, an additional linker flag
should be specified in your app project to force load the select TF ops
framework. In your Xcode project, go to `Build Settings` -> `Other Linker
Flags`, and add:
```text
-force_load <path/to/your/TensorFlowLiteSelectTfOps.framework/TensorFlowLiteSelectTfOps>
```
### C/C++
If you're using Bazel or
[CMake](https://www.tensorflow.org/lite/guide/build_cmake) to build TensorFlow
Lite interpreter, you can enable Flex delegate by linking a TensorFlow Lite Flex
delegate shared library. You can build it with Bazel as the following command.
```
bazel build -c opt --config=monolithic tensorflow/lite/delegates/flex:tensorflowlite_flex
```
This command generates the following shared library in
`bazel-bin/tensorflow/lite/delegates/flex`.
Platform | Library name
-------- | ------------------------------
Linux | `libtensorflowlite_flex.so`
macOS | `libtensorflowlite_flex.dylib`
Windows | `tensorflowlite_flex.dll`
Note that the necessary `TfLiteDelegate` will be installed automatically when
creating the interpreter at runtime as long as the shared library is linked. It
is not necessary to explicitly install the delegate instance as is typically
required with other delegate types.
**Note:** This feature is available since version 2.7.
### Python
TensorFlow Lite with select TensorFlow ops will be installed automatically with
the [TensorFlow pip package](https://www.tensorflow.org/install/pip). You can
also choose to only install the
[TensorFlow Lite Interpreter pip package](https://www.tensorflow.org/lite/guide/python#install_just_the_tensorflow_lite_interpreter).
Note: TensorFlow Lite with select TensorFlow ops are available in the TensorFlow
pip package version since 2.3 for Linux and 2.4 for other environments.
## Metrics
### Performance
When using a mixture of both builtin and select TensorFlow ops, all of the same
TensorFlow Lite optimizations and optimized builtin ops will be available and
usable with the converted model.
The following table describes the average time taken to run inference on
MobileNet on a Pixel 2. The listed times are an average of 100 runs. These
targets were built for Android using the flags: `--config=android_arm64 -c opt`.
Build | Time (milliseconds)
------------------------------------ | -------------------
Only built-in ops (`TFLITE_BUILTIN`) | 260.7
Using only TF ops (`SELECT_TF_OPS`) | 264.5
### Binary size
The following table describes the binary size of TensorFlow Lite for each build.
These targets were built for Android using `--config=android_arm -c opt`.
Build | C++ Binary Size | Android APK Size
------------------------- | --------------- | ----------------
Only built-in ops | 796 KB | 561 KB
Built-in ops + TF ops | 23.0 MB | 8.0 MB
Built-in ops + TF ops (1) | 4.1 MB | 1.8 MB
(1) These libraries are selectively built for
[i3d-kinetics-400 model](https://tfhub.dev/deepmind/i3d-kinetics-400/1) with 8
TFLite builtin ops and 3 Tensorflow ops. For more details, please see the
[Reduce TensorFlow Lite binary size](../guide/reduce_binary_size.md) section.
## Known limitations
* Unsupported types: Certain TensorFlow ops may not support the full set of
input/output types that are typically available in TensorFlow.
## Updates
* Version 2.6
- Supports for GraphDef-attribute based operators and HashTable resource
initializations have improved.
* Version 2.5
- You can apply an optimization known as
[post training quantization](../performance/post_training_quantization.md)
* Version 2.4
- Compatibility with hardware accelerated delegates has improved
tensorflow/lite/g3doc/guide/ops_version.md
+242
View File
@@ -0,0 +1,242 @@
# TensorFlow Lite operator versions
This document describes TensorFlow Lite's op versioning schema. Op versioning
enables developers to add new functionalities and parameters into existing ops.
In addition, it guarantees the following:
* Backward compatibility: New TensorFlow Lite implementation should handle an
old model file.
* Forward compatibility: Old TensorFlow Lite implementation should handle a
new model file produced by new version of converter, as long as no new
features are used.
* Forward in-compatibility detection: If an old TensorFlow Lite implementation
reads a new model that contains a new version of an op which isn't
supported, it should report the error.
## Example: Adding dilation into depthwise convolution
The remainder of this document explains op versioning in TFLite by showing how
to add dilation parameters to the depthwise convolution operation.
Knowledge of dilation is not required to understand this document. Note that:
* 2 new integer parameters will be added: `dilation_width_factor` and
`dilation_height_factor`.
* Old depthwise convolution kernels that don't support dilation are equivalent
to setting the dilation factors to 1.
### Change FlatBuffer schema
To add new parameters into an op, change the options table in
`lite/schema/schema.fbs`.
For example, the options table of depthwise convolution looks like this:
```
table DepthwiseConv2DOptions {
padding:Padding;
stride_w:int;
stride_h:int;
depth_multiplier:int;
fused_activation_function:ActivationFunctionType;
}
```
When adding new parameters:
* Add comments indicating which parameters are supported by which version.
* When the new implementation gets the default values for newly added
parameters, it should work exactly the same as the old implementation.
The table will be like this after the new parameters are added:
```
table DepthwiseConv2DOptions {
// Parameters for DepthwiseConv version 1 or above.
padding:Padding;
stride_w:int;
stride_h:int;
depth_multiplier:int;
fused_activation_function:ActivationFunctionType;
// Parameters for DepthwiseConv version 2 or above.
dilation_w_factor:int = 1;
dilation_h_factor:int = 1;
}
```
The file `lite/schema/schema_generated.h` should be re-generated for the new
schema.
### Change C structures and kernel implementation
In TensorFlow Lite, the kernel implementation is decoupled from FlatBuffer
definition. The kernels read the parameter from C structures defined in
`lite/c/builtin_op_data.h`.
The original depthwise convolution parameter is as follows:
```
typedef struct {
TfLitePadding padding;
int stride_width;
int stride_height;
int depth_multiplier;
TfLiteFusedActivation activation;
} TfLiteDepthwiseConvParams;
```
As with the FlatBuffer schema, add comments indicating which parameters are
supported starting from which version. The result is seen below:
```
typedef struct {
// Parameters for DepthwiseConv version 1 or above.
TfLitePadding padding;
int stride_width;
int stride_height;
int depth_multiplier;
TfLiteFusedActivation activation;
// Parameters for DepthwiseConv version 2 or above.
int dilation_width_factor;
int dilation_height_factor;
} TfLiteDepthwiseConvParams;
```
Please also change the kernel implementation to read the newly added parameters
from the C structures. The details are omitted here.
### Change the FlatBuffer reading code
The logic to read FlatBuffer and produce C structure is in
`lite/core/api/flatbuffer_conversions.cc`.
Update the file to handle the new parameters, as shown below:
```
TfLiteStatus ParseDepthwiseConv2D(const Operator* op,
ErrorReporter* error_reporter,
BuiltinDataAllocator* allocator,
void** builtin_data) {
CheckParsePointerParams(op, error_reporter, allocator, builtin_data);
SafeBuiltinDataAllocator safe_allocator(allocator);
std::unique_ptr<TfLiteDepthwiseConvParams,
SafeBuiltinDataAllocator::BuiltinDataDeleter>
params = safe_allocator.Allocate<TfLiteDepthwiseConvParams>();
TF_LITE_ENSURE(error_reporter, params != nullptr);
const DepthwiseConv2DOptions* schema_params =
op->builtin_options_as_DepthwiseConv2DOptions();
if (schema_params != nullptr) {
params->padding = ConvertPadding(schema_params->padding());
params->stride_width = schema_params->stride_w();
params->stride_height = schema_params->stride_h();
params->depth_multiplier = schema_params->depth_multiplier();
params->activation =
ConvertActivation(schema_params->fused_activation_function());
params->dilation_width_factor = schema_params->dilation_w_factor();
params->dilation_height_factor = schema_params->dilation_h_factor();
}
*builtin_data = params.release();
return kTfLiteOk;
}
```
It's not required to check the op version here. When the new implementation
reads an old model file where dilation factors are missing, it will use 1 as the
default value, and the new kernel will work consistently with the old kernel.
### Change kernel registration
The MutableOpResolver (defined in `lite/mutable_op_resolver.h`) provides a few
functions to register op kernels. The minimum and maximum version are 1 by
default:
```
void AddBuiltin(tflite::BuiltinOperator op, TfLiteRegistration* registration,
int min_version = 1, int max_version = 1);
void AddCustom(const char* name, TfLiteRegistration* registration,
int min_version = 1, int max_version = 1);
```
The built-in ops are registered in `lite/kernels/register.cc`. In this example,
we implemented a new op kernel which can handle `DepthwiseConv2D` version 1 and
2, so we need to change this line:
```
AddBuiltin(BuiltinOperator_DEPTHWISE_CONV_2D, Register_DEPTHWISE_CONV_2D());
```
to:
```
AddBuiltin(BuiltinOperator_DEPTHWISE_CONV_2D, Register_DEPTHWISE_CONV_2D(),
/* min_version = */ 1,
/* max_version = */ 2);
```
### Change TFLite op version
The next step is to make TFLite populate the minimum version that's required to
execute the op. In this example, it means:
* Populate version=1 when dilation factors are all 1.
* Populate version=2 otherwise.
Modify `GetBuiltinOperatorVersion` function for the operator in
`lite/tools/versioning/op_version.cc` by adding the new version to the case of
`DepthwiseConv2D`:
```
case BuiltinOperator_DEPTHWISE_CONV_2D:
auto depthwise_conv_params =
reinterpret_cast<TfLiteDepthwiseConvParams*>(op_sig.builtin_data);
TFLITE_DCHECK(depthwise_conv_params != nullptr);
if (depthwise_conv_params->dilation_width_factor != 1 ||
depthwise_conv_params->dilation_height_factor != 1) {
return 2;
}
return 1;
```
### Update the operator version map
The last step is to add the new version info into the operator version map. This
step is required because we need to generate the model's minimum required
runtime version based on this version map.
To do this, you need to add a new map entry in
`lite/tools/versioning/runtime_version.cc`.
In this example, you need to add the following entry into `op_version_map`:
```
{{BuiltinOperator_DEPTHWISE_CONV_2D, 2}, %CURRENT_RUNTIME_VERSION%}
```
where `%CURRENT_RUNTIME_VERSION%` corresponds to the current runtime version
defined in [tensorflow/core/public/version.h](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/public/version.h).
### Delegation implementation
TensorFlow Lite provides a delegation API which enables delegating ops to
hardware backends. In the delegate's `Prepare` function, check if the version is
supported for every node in Delegation code.
```
const int kMaxVersion = 1;
TfLiteNode* node;
TfLiteRegistration* registration = nullptr;
TF_LITE_ENSURE_STATUS(context->GetNodeAndRegistration(context, node_index, &node, &registration));
if (registration->version > kMaxVersion) {
// Reject the node if the version isn't supported.
}
```
This is required even if the delegation only supports version 1 ops, so the
delegation can detect incompatibility when getting a higher version op.
tensorflow/lite/g3doc/guide/python.md
+118
View File
@@ -0,0 +1,118 @@
# Quickstart for Linux-based devices with Python
Using TensorFlow Lite with Python is great for embedded devices based on Linux,
such as [Raspberry Pi](https://www.raspberrypi.org/){:.external} and
[Coral devices with Edge TPU](https://coral.withgoogle.com/){:.external},
among many others.
This page shows how you can start running TensorFlow Lite models with Python in
just a few minutes. All you need is a TensorFlow model [converted to TensorFlow
Lite](../models/convert/). (If you don't have a model converted yet, you can
experiment using the model provided with the example linked below.)
## About the TensorFlow Lite runtime package
To quickly start executing TensorFlow Lite models with Python, you can install
just the TensorFlow Lite interpreter, instead of all TensorFlow packages. We
call this simplified Python package `tflite_runtime`.
The `tflite_runtime` package is a fraction the size of the full `tensorflow`
package and includes the bare minimum code required to run inferences with
TensorFlow Lite—primarily the
[`Interpreter`](https://www.tensorflow.org/api_docs/python/tf/lite/Interpreter)
Python class. This small package is ideal when all you want to do is execute
`.tflite` models and avoid wasting disk space with the large TensorFlow library.
Note: If you need access to other Python APIs, such as the
[TensorFlow Lite Converter](../models/convert/), you must install the
[full TensorFlow package](https://www.tensorflow.org/install/).
For example, the [Select TF ops]
(https://www.tensorflow.org/lite/guide/ops_select) are not included in the
`tflite_runtime` package. If your models have any dependencies to the Select TF
ops, you need to use the full TensorFlow package instead.
## Install TensorFlow Lite for Python
You can install on Linux with pip:
<pre class="devsite-terminal devsite-click-to-copy">
python3 -m pip install tflite-runtime
</pre>
## Supported platforms
The `tflite-runtime` Python wheels are pre-built and provided for these
platforms:
* Linux armv7l (e.g. Raspberry Pi 2, 3, 4 and Zero 2 running Raspberry Pi OS
32-bit)
* Linux aarch64 (e.g. Raspberry Pi 3, 4 running Debian ARM64)
* Linux x86_64
If you want to run TensorFlow Lite models on other platforms, you should either
use the [full TensorFlow package](https://www.tensorflow.org/install/), or
[build the tflite-runtime package from source](build_cmake_pip.md).
If you're using TensorFlow with the Coral Edge TPU, you should
instead follow the appropriate [Coral setup documentation](https://coral.ai/docs/setup).
Note: We no longer update the Debian package `python3-tflite-runtime`. The
latest Debian package is for TF version 2.5, which you can install by following
[these older instructions](https://github.com/tensorflow/tensorflow/blob/v2.5.0/tensorflow/lite/g3doc/guide/python.md#install-tensorflow-lite-for-python).
Note: We no longer release pre-built `tflite-runtime` wheels for Windows and
macOS. For these platforms, you should use the
[full TensorFlow package](https://www.tensorflow.org/install/), or
[build the tflite-runtime package from source](build_cmake_pip.md).
## Run an inference using tflite_runtime
Instead of importing `Interpreter` from the `tensorflow` module, you now need to
import it from `tflite_runtime`.
For example, after you install the package above, copy and run the
[`label_image.py`](
https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/examples/python/)
file. It will (probably) fail because you don't have the `tensorflow` library
installed. To fix it, edit this line of the file:
```python
import tensorflow as tf
```
So it instead reads:
```python
import tflite_runtime.interpreter as tflite
```
And then change this line:
```python
interpreter = tf.lite.Interpreter(model_path=args.model_file)
```
So it reads:
```python
interpreter = tflite.Interpreter(model_path=args.model_file)
```
Now run `label_image.py` again. That's it! You're now executing TensorFlow Lite
models.
## Learn more
* For more details about the `Interpreter` API, read
[Load and run a model in Python](inference.md#load-and-run-a-model-in-python).
* If you have a Raspberry Pi, check out a [video series](https://www.youtube.com/watch?v=mNjXEybFn98&list=PLQY2H8rRoyvz_anznBg6y3VhuSMcpN9oe)
about how to run object detection on Raspberry Pi using TensorFlow Lite.
* If you're using a Coral ML accelerator, check out the
[Coral examples on GitHub](https://github.com/google-coral/tflite/tree/master/python/examples).
* To convert other TensorFlow models to TensorFlow Lite, read about the
[TensorFlow Lite Converter](../models/convert/).
* If you want to build `tflite_runtime` wheel, read
[Build TensorFlow Lite Python Wheel Package](build_cmake_pip.md)
tensorflow/lite/g3doc/guide/reduce_binary_size.md
+429
View File
@@ -0,0 +1,429 @@
# Reduce TensorFlow Lite binary size
## Overview
When deploying models for on-device machine learning (ODML) applications, it is
important to be aware of the limited memory that is available on mobile devices.
Model binary sizes are closely correlated to the number of ops used in the
model. TensorFlow Lite enables you to reduce model binary sizes by using
selective builds. Selective builds skip unused operations in your model set and
produce a compact library with just the runtime and the op kernels required for
the model to run on your mobile device.
Selective build applies on the following three operations libraries.
1. [TensorFlow Lite built-in ops library](https://www.tensorflow.org/lite/guide/ops_compatibility)
1. [TensorFlow Lite custom ops](https://www.tensorflow.org/lite/guide/ops_custom)
1. [Select TensorFlow ops library](https://www.tensorflow.org/lite/guide/ops_select)
The table below demonstrates the impact of selective builds for some common use
cases:
<table>
<thead>
<tr>
<th>Model Name</th>
<th>Domain</th>
<th>Target architecture</th>
<th>AAR file size(s)</th>
</tr>
</thead>
<tr>
<td rowspan = 2>
<a href="https://storage.googleapis.com/download.tensorflow.org/models/mobilenet_v1_2018_08_02/mobilenet_v1_1.0_224.tgz">Mobilenet_1.0_224(float)</a>
</td>
<td rowspan = 2>Image classification</td>
<td>armeabi-v7a</td>
<td>tensorflow-lite.aar (296,635 bytes)</td>
</tr>
<tr>
<td>arm64-v8a</td>
<td>tensorflow-lite.aar (382,892 bytes)</td>
</tr>
<tr>
<td rowspan = 2>
<a href="https://tfhub.dev/google/lite-model/spice/">SPICE</a>
</td>
<td rowspan = 2>Sound pitch extraction</td>
<td>armeabi-v7a</td>
<td>tensorflow-lite.aar (375,813 bytes)<br />tensorflow-lite-select-tf-ops.aar (1,676,380 bytes)</td>
</tr>
<tr>
<td>arm64-v8a</td>
<td>tensorflow-lite.aar (421,826 bytes)<br />tensorflow-lite-select-tf-ops.aar (2,298,630 bytes)</td>
</tr>
<tr>
<td rowspan = 2>
<a href="https://tfhub.dev/deepmind/i3d-kinetics-400/1">i3d-kinetics-400</a>
</td>
<td rowspan = 2>Video classification</td>
<td>armeabi-v7a</td>
<td>tensorflow-lite.aar (240,085 bytes)<br />tensorflow-lite-select-tf-ops.aar (1,708,597 bytes)</td>
</tr>
<tr>
<td>arm64-v8a</td>
<td>tensorflow-lite.aar (273,713 bytes)<br />tensorflow-lite-select-tf-ops.aar (2,339,697 bytes)</td>
</tr>
</table>
Note: This feature is currently experimental and available since version 2.4 and
may change.
## Selectively build TensorFlow Lite with Bazel
This section assumes that you have downloaded TensorFlow source codes and
[set up the local development environment](https://www.tensorflow.org/lite/android/lite_build#set_up_build_environment_without_docker)
to Bazel.
### Build AAR files for Android project
You can build the custom TensorFlow Lite AARs by providing your model file paths
as follows.
```sh
sh tensorflow/lite/tools/build_aar.sh \
--input_models=/a/b/model_one.tflite,/c/d/model_two.tflite \
--target_archs=x86,x86_64,arm64-v8a,armeabi-v7a
```
The above command will generate the AAR file `bazel-bin/tmp/tensorflow-lite.aar`
for TensorFlow Lite built-in and custom ops; and optionally, generates the aar
file `bazel-bin/tmp/tensorflow-lite-select-tf-ops.aar` if your models contain
Select TensorFlow ops. Note that this builds a "fat" AAR with several different
architectures; if you don't need all of them, use the subset appropriate for
your deployment environment.
### Build with custom ops
If you have developed Tensorflow Lite models with custom ops, you can build them
by adding the following flags to the build command:
```sh
sh tensorflow/lite/tools/build_aar.sh \
--input_models=/a/b/model_one.tflite,/c/d/model_two.tflite \
--target_archs=x86,x86_64,arm64-v8a,armeabi-v7a \
--tflite_custom_ops_srcs=/e/f/file1.cc,/g/h/file2.h \
--tflite_custom_ops_deps=dep1,dep2
```
The `tflite_custom_ops_srcs` flag contains source files of your custom ops and
the `tflite_custom_ops_deps` flag contains dependencies to build those source
files. Note that these dependencies must exist in the TensorFlow repo.
### Advanced Usages: Custom Bazel rules
If your project is using Bazel and you would like to define custom TFLite
dependencies for a given set of models, you can define following rule(s) in your
project repository:
For the models with the builtin ops only:
```bazel
load(
"@org_tensorflow//tensorflow/lite:build_def.bzl",
"tflite_custom_android_library",
"tflite_custom_c_library",
"tflite_custom_cc_library",
)
# A selectively built TFLite Android library.
tflite_custom_android_library(
name = "selectively_built_android_lib",
models = [
":model_one.tflite",
":model_two.tflite",
],
)
# A selectively built TFLite C library.
tflite_custom_c_library(
name = "selectively_built_c_lib",
models = [
":model_one.tflite",
":model_two.tflite",
],
)
# A selectively built TFLite C++ library.
tflite_custom_cc_library(
name = "selectively_built_cc_lib",
models = [
":model_one.tflite",
":model_two.tflite",
],
)
```
For the models with the [Select TF ops](../guide/ops_select.md):
```bazel
load(
"@org_tensorflow//tensorflow/lite/delegates/flex:build_def.bzl",
"tflite_flex_android_library",
"tflite_flex_cc_library",
)
# A Select TF ops enabled selectively built TFLite Android library.
tflite_flex_android_library(
name = "selective_built_tflite_flex_android_lib",
models = [
":model_one.tflite",
":model_two.tflite",
],
)
# A Select TF ops enabled selectively built TFLite C++ library.
tflite_flex_cc_library(
name = "selective_built_tflite_flex_cc_lib",
models = [
":model_one.tflite",
":model_two.tflite",
],
)
```
### Advanced Usages: Build custom C/C++ shared libraries
If you would like to build your own custom TFLite C/C++ shared objects towards
the given models, you can follow the below steps:
Create a temporary BUILD file by running the following command at the root
directory of the TensorFlow source code:
```sh
mkdir -p tmp && touch tmp/BUILD
```
#### Building custom C shared objects
If you would like to build a custom TFLite C shared object, add the following to
`tmp/BUILD` file:
```bazel
load(
"//tensorflow/lite:build_def.bzl",
"tflite_custom_c_library",
"tflite_cc_shared_object",
)
tflite_custom_c_library(
name = "selectively_built_c_lib",
models = [
":model_one.tflite",
":model_two.tflite",
],
)
# Generates a platform-specific shared library containing the TensorFlow Lite C
# API implementation as define in `c_api.h`. The exact output library name
# is platform dependent:
# - Linux/Android: `libtensorflowlite_c.so`
# - Mac: `libtensorflowlite_c.dylib`
# - Windows: `tensorflowlite_c.dll`
tflite_cc_shared_object(
name = "tensorflowlite_c",
linkopts = select({
"//tensorflow:ios": [
"-Wl,-exported_symbols_list,$(location //tensorflow/lite/c:exported_symbols.lds)",
],
"//tensorflow:macos": [
"-Wl,-exported_symbols_list,$(location //tensorflow/lite/c:exported_symbols.lds)",
],
"//tensorflow:windows": [],
"//conditions:default": [
"-z defs",
"-Wl,--version-script,$(location //tensorflow/lite/c:version_script.lds)",
],
}),
per_os_targets = True,
deps = [
":selectively_built_c_lib",
"//tensorflow/lite/c:exported_symbols.lds",
"//tensorflow/lite/c:version_script.lds",
],
)
```
The newly added target can be built as follows:
```sh
bazel build -c opt --cxxopt=--std=c++17 \
//tmp:tensorflowlite_c
```
and for Android (replace `android_arm` with `android_arm64` for 64-bit):
```sh
bazel build -c opt --cxxopt=--std=c++17 --config=android_arm \
//tmp:tensorflowlite_c
```
#### Building custom C++ shared objects
If you would like to build a custom TFLite C++ shared object, add the following
to `tmp/BUILD` file:
```bazel
load(
"//tensorflow/lite:build_def.bzl",
"tflite_custom_cc_library",
"tflite_cc_shared_object",
)
tflite_custom_cc_library(
name = "selectively_built_cc_lib",
models = [
":model_one.tflite",
":model_two.tflite",
],
)
# Shared lib target for convenience, pulls in the core runtime and builtin ops.
# Note: This target is not yet finalized, and the exact set of exported (C/C++)
# APIs is subject to change. The output library name is platform dependent:
# - Linux/Android: `libtensorflowlite.so`
# - Mac: `libtensorflowlite.dylib`
# - Windows: `tensorflowlite.dll`
tflite_cc_shared_object(
name = "tensorflowlite",
# Until we have more granular symbol export for the C++ API on Windows,
# export all symbols.
features = ["windows_export_all_symbols"],
linkopts = select({
"//tensorflow:macos": [
"-Wl,-exported_symbols_list,$(location //tensorflow/lite:tflite_exported_symbols.lds)",
],
"//tensorflow:windows": [],
"//conditions:default": [
"-Wl,-z,defs",
"-Wl,--version-script,$(location //tensorflow/lite:tflite_version_script.lds)",
],
}),
per_os_targets = True,
deps = [
":selectively_built_cc_lib",
"//tensorflow/lite:tflite_exported_symbols.lds",
"//tensorflow/lite:tflite_version_script.lds",
],
)
```
The newly added target can be built as follows:
```sh
bazel build -c opt --cxxopt=--std=c++17 \
//tmp:tensorflowlite
```
and for Android (replace `android_arm` with `android_arm64` for 64-bit):
```sh
bazel build -c opt --cxxopt=--std=c++17 --config=android_arm \
//tmp:tensorflowlite
```
For the models with the Select TF ops, you also need to build the following
shared library as well:
```bazel
load(
"@org_tensorflow//tensorflow/lite/delegates/flex:build_def.bzl",
"tflite_flex_shared_library"
)
# Shared lib target for convenience, pulls in the standard set of TensorFlow
# ops and kernels. The output library name is platform dependent:
# - Linux/Android: `libtensorflowlite_flex.so`
# - Mac: `libtensorflowlite_flex.dylib`
# - Windows: `libtensorflowlite_flex.dll`
tflite_flex_shared_library(
name = "tensorflowlite_flex",
models = [
":model_one.tflite",
":model_two.tflite",
],
)
```
The newly added target can be built as follows:
```sh
bazel build -c opt --cxxopt='--std=c++17' \
--config=monolithic \
--host_crosstool_top=@bazel_tools//tools/cpp:toolchain \
//tmp:tensorflowlite_flex
```
and for Android (replace `android_arm` with `android_arm64` for 64-bit):
```sh
bazel build -c opt --cxxopt='--std=c++17' \
--config=android_arm \
--config=monolithic \
--host_crosstool_top=@bazel_tools//tools/cpp:toolchain \
//tmp:tensorflowlite_flex
```
## Selectively Build TensorFlow Lite with Docker
This section assumes that you have installed
[Docker](https://docs.docker.com/get-docker/) on your local machine and
downloaded the TensorFlow Lite Dockerfile
[here](https://www.tensorflow.org/lite/android/lite_build#set_up_build_environment_using_docker).
After downloading the above Dockerfile, you can build the docker image by
running:
```shell
docker build . -t tflite-builder -f tflite-android.Dockerfile
```
### Build AAR files for Android project
Download the script for building with Docker by running:
```sh
curl -o build_aar_with_docker.sh \
https://raw.githubusercontent.com/tensorflow/tensorflow/master/tensorflow/lite/tools/build_aar_with_docker.sh &&
chmod +x build_aar_with_docker.sh
```
Then, you can build the custom TensorFlow Lite AAR by providing your model file
paths as follows.
```sh
sh build_aar_with_docker.sh \
--input_models=/a/b/model_one.tflite,/c/d/model_two.tflite \
--target_archs=x86,x86_64,arm64-v8a,armeabi-v7a \
--checkpoint=master \
[--cache_dir=<path to cache directory>]
```
The `checkpoint` flag is a commit, a branch or a tag of the TensorFlow repo that
you want to checkout before building the libraries; by default it is the latest
release branch. The above command will generate the AAR file
`tensorflow-lite.aar` for TensorFlow Lite built-in and custom ops and optionally
the AAR file `tensorflow-lite-select-tf-ops.aar` for Select TensorFlow ops in
your current directory.
The --cache_dir specify the cache directory. If not provided, the script will
create a directory named `bazel-build-cache` under current working directory for
caching.
## Add AAR files to project
Add AAR files by directly
[importing the AAR into your project](https://www.tensorflow.org/lite/android/lite_build#add_aar_directly_to_project),
or by
[publishing the custom AAR to your local Maven repository](https://www.tensorflow.org/lite/android/lite_build#install_aar_to_local_maven_repository).
Note that you have to add the AAR files for `tensorflow-lite-select-tf-ops.aar`
as well if you generate it.
## Selective Build for iOS
Please see the
[Building locally section](../guide/build_ios.md#building_locally) to set up the
build environment and configure TensorFlow workspace and then follow the
[guide](../guide/build_ios.md#selectively_build_tflite_frameworks) to use the
selective build script for iOS.
tensorflow/lite/g3doc/guide/roadmap.md
+105
View File
@@ -0,0 +1,105 @@
# TensorFlow Lite Roadmap
**Updated: May, 2021**
The following represents a high level overview of our roadmap. You should be
aware that this roadmap may change at any time and the order below does not
reflect any type of priority.
We break our roadmap into four key segments: usability, performance,
optimization and portability. We strongly encourage you to comment on our
roadmap and provide us feedback in the
[TensorFlow Lite discussion group](https://groups.google.com/a/tensorflow.org/g/tflite).
## Usability
* **Expanded ops coverage**
* Add targeted ops based on user feedback.
* Add targeted op sets for specific domains and areas including Random
ops, base Keras layer ops, hash tables, select training ops.
* **More assistive tooling**
* Provide TensorFlow graph annotations and compatibility tools to validate
TFLite and hardware accelerator compatibility during-training and
after-conversion.
* Allow targeting and optimizing for specific accelerators during
conversion.
* **On-device training**
* Support on-device training for personalization and transfer learning,
including a Colab demonstrating end-to-end usage.
* Support variable/resource types (both for inference and training)
* Support converting and executing graphs with multiple function (or
signature) entry-points.
* **Enhanced Android Studio integration**
* Drag and drop TFLite models into Android Studio to generate model
interfaces.
* Improve Android Studio profiling support, including memory profiling.
* **Model Maker**
* Support newer tasks, including object detection, recommendation, and
audio classification, covering a wide collection of common usage.
* Support more data sets to make transfer learning easier.
* **Task Library**
* Support more model types (e.g. audio, NLP) with associated pre and post
processing capabilities.
* Update more reference examples with Task APIs.
* Support out-of-the-box acceleration for all tasks.
* **More SOTA models and examples**
* Add more examples (e.g. audio, NLP, structure-data related) to
demonstrate model usage as well as new features and APIs, covering
different platforms.
* Create shareable backbone models for on-device to reduce training and
deployment costs.
* **Seamless deployment across multiple platforms**
* Run TensorFlow Lite models on the web.
* **Improved cross-platform support**
* Extend and improve APIs for Java on Android, Swift on iOS, Python on
RPi.
* Enhance CMake support (e.g., broader accelerator support).
* **Better frontend support**
* Improve compatibility with various authoring frontends, including Keras,
tf.numpy.
## Performance
* **Better tooling**
* Public dashboard for tracking performance gains with each release.
* Tooling for better understanding graph compatibility with target
accelerators.
* **Improved CPU performance**
* XNNPack enabled by default for faster floating point inference.
* End-to-end half precision (float16) support with optimized kernels.
* **Updated NN API support**
* Full support for newer Android version NN API features, ops, and types.
* **GPU optimizations**
* Improved startup time with delegate serialization support.
* Hardware buffer interop for zero-copy inference.
* Wider availability of on device acceleration.
* Better op coverage.
## Optimization
* **Quantization**
* Selective post-training quantization to exclude certain layers from
quantization.
* Quantization debugger to inspect quantization error losses per each
layer.
* Applying quantization-aware training on more model coverage e.g.
TensorFlow Model Garden.
* Quality and performance improvements for post-training dynamic-range
quantization.
* Tensor Compression API to allow compression algorithms such as SVD.
* **Pruning / sparsity**
* Combine configurable training-time (pruning + quantization-aware
training) APIs.
* Increase sparity application on TF Model Garden models.
* Sparse model execution support in TensorFlow Lite.
## Portability
* **Microcontroller Support**
* Add support for a range of 32-bit MCU architecture use cases for speech
and image classification.
* Audio Frontend: In-graph audio pre-processing and acceleration support
* Sample code and models for vision and audio data.
tensorflow/lite/g3doc/guide/signatures.ipynb
+501
View File
@@ -0,0 +1,501 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "g_nWetWWd_ns",
"metadata": {
"id": "g_nWetWWd_ns"
},
"source": [
"##### Copyright 2024 The AI Edge Authors."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2pHVBk_seED1",
"metadata": {
"cellView": "form",
"id": "2pHVBk_seED1"
},
"outputs": [],
"source": [
"#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"id": "M7vSdG6sAIQn",
"metadata": {
"id": "M7vSdG6sAIQn"
},
"source": [
"# Signatures in TensorFlow Lite"
]
},
{
"cell_type": "markdown",
"id": "fwc5GKHBASdc",
"metadata": {
"id": "fwc5GKHBASdc"
},
"source": [
"<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
" <td>\n",
" <a target=\"_blank\" href=\"https://www.tensorflow.org/lite/guide/signatures\"><img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" />View on TensorFlow.org</a>\n",
" </td>\n",
" <td>\n",
" <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/guide/signatures.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
" </td>\n",
" <td>\n",
" <a target=\"_blank\" href=\"https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/guide/signatures.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://storage.googleapis.com/tensorflow_docs/tensorflow/tensorflow/lite/g3doc/guide/signatures.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\" />Download notebook</a>\n",
" </td>\n",
"</table>"
]
},
{
"cell_type": "markdown",
"id": "9ee074e4",
"metadata": {
"id": "9ee074e4"
},
"source": [
"TensorFlow Lite supports converting TensorFlow model's input/output\n",
"specifications to TensorFlow Lite models. The input/output specifications are\n",
"called \"signatures\". Signatures can be specified when building a SavedModel or\n",
"creating concrete functions.\n",
"\n",
"Signatures in TensorFlow Lite provide the following features:\n",
"\n",
"* They specify inputs and outputs of the converted TensorFlow Lite model by\n",
" respecting the TensorFlow model's signatures.\n",
"* Allow a single TensorFlow Lite model to support multiple entry points.\n",
"\n",
"The signature is composed of three pieces:\n",
"\n",
"* Inputs: Map for inputs from input name in the signature to an input tensor.\n",
"* Outputs: Map for output mapping from output name in signature to an output\n",
" tensor.\n",
"* Signature Key: Name that identifies an entry point of the graph.\n",
"\n"
]
},
{
"cell_type": "markdown",
"id": "UaWdLA3fQDK2",
"metadata": {
"id": "UaWdLA3fQDK2"
},
"source": [
"## Setup"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9j4MGqyKQEo4",
"metadata": {
"id": "9j4MGqyKQEo4"
},
"outputs": [],
"source": [
"import tensorflow as tf"
]
},
{
"cell_type": "markdown",
"id": "FN2N6hPEP-Ay",
"metadata": {
"id": "FN2N6hPEP-Ay"
},
"source": [
"## Example model\n",
"\n",
"Let's say we have two tasks, e.g., encoding and decoding, as a TensorFlow model:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d8577c80",
"metadata": {
"id": "d8577c80"
},
"outputs": [],
"source": [
"class Model(tf.Module):\n",
"\n",
" @tf.function(input_signature=[tf.TensorSpec(shape=[None], dtype=tf.float32)])\n",
" def encode(self, x):\n",
" result = tf.strings.as_string(x)\n",
" return {\n",
" \"encoded_result\": result\n",
" }\n",
"\n",
" @tf.function(input_signature=[tf.TensorSpec(shape=[None], dtype=tf.string)])\n",
" def decode(self, x):\n",
" result = tf.strings.to_number(x)\n",
" return {\n",
" \"decoded_result\": result\n",
" }"
]
},
{
"cell_type": "markdown",
"id": "9c814c6e",
"metadata": {
"id": "9c814c6e"
},
"source": [
"In the signature wise, the above TensorFlow model can be summarized as follows:\n",
"\n",
"* Signature\n",
"\n",
" - Key: encode\n",
" - Inputs: {\"x\"}\n",
" - Output: {\"encoded_result\"}\n",
"\n",
"* Signature\n",
"\n",
" - Key: decode\n",
" - Inputs: {\"x\"}\n",
" - Output: {\"decoded_result\"}"
]
},
{
"cell_type": "markdown",
"id": "c4099f20",
"metadata": {
"id": "c4099f20"
},
"source": [
"## Convert a model with Signatures\n",
"\n",
"TensorFlow Lite converter APIs will bring the above signature information into\n",
"the converted TensorFlow Lite model.\n",
"\n",
"This conversion functionality is available on all the converter APIs starting\n",
"from TensorFlow version 2.7.0. See example usages.\n"
]
},
{
"cell_type": "markdown",
"id": "Qv0WwFQkQgnO",
"metadata": {
"id": "Qv0WwFQkQgnO"
},
"source": [
"\n",
"### From Saved Model"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "96c8fc79",
"metadata": {
"id": "96c8fc79"
},
"outputs": [],
"source": [
"model = Model()\n",
"\n",
"# Save the model\n",
"SAVED_MODEL_PATH = 'content/saved_models/coding'\n",
"\n",
"tf.saved_model.save(\n",
" model, SAVED_MODEL_PATH,\n",
" signatures={\n",
" 'encode': model.encode.get_concrete_function(),\n",
" 'decode': model.decode.get_concrete_function()\n",
" })\n",
"\n",
"# Convert the saved model using TFLiteConverter\n",
"converter = tf.lite.TFLiteConverter.from_saved_model(SAVED_MODEL_PATH)\n",
"converter.target_spec.supported_ops = [\n",
" tf.lite.OpsSet.TFLITE_BUILTINS, # enable TensorFlow Lite ops.\n",
" tf.lite.OpsSet.SELECT_TF_OPS # enable TensorFlow ops.\n",
"]\n",
"tflite_model = converter.convert()\n",
"\n",
"# Print the signatures from the converted model\n",
"interpreter = tf.lite.Interpreter(model_content=tflite_model)\n",
"signatures = interpreter.get_signature_list()\n",
"print(signatures)"
]
},
{
"cell_type": "markdown",
"id": "5baa9f17",
"metadata": {
"id": "5baa9f17"
},
"source": [
"### From Keras Model"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "71f29229",
"metadata": {
"id": "71f29229"
},
"outputs": [],
"source": [
"# Generate a Keras model.\n",
"keras_model = tf.keras.Sequential(\n",
" [\n",
" tf.keras.layers.Dense(2, input_dim=4, activation='relu', name='x'),\n",
" tf.keras.layers.Dense(1, activation='relu', name='output'),\n",
" ]\n",
")\n",
"\n",
"# Convert the keras model using TFLiteConverter.\n",
"# Keras model converter API uses the default signature automatically.\n",
"converter = tf.lite.TFLiteConverter.from_keras_model(keras_model)\n",
"tflite_model = converter.convert()\n",
"\n",
"# Print the signatures from the converted model\n",
"interpreter = tf.lite.Interpreter(model_content=tflite_model)\n",
"\n",
"signatures = interpreter.get_signature_list()\n",
"print(signatures)"
]
},
{
"cell_type": "markdown",
"id": "e4d30f85",
"metadata": {
"id": "e4d30f85"
},
"source": [
"### From Concrete Functions"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c9e8a742",
"metadata": {
"id": "c9e8a742"
},
"outputs": [],
"source": [
"model = Model()\n",
"\n",
"# Convert the concrete functions using TFLiteConverter\n",
"converter = tf.lite.TFLiteConverter.from_concrete_functions(\n",
" [model.encode.get_concrete_function(),\n",
" model.decode.get_concrete_function()], model)\n",
"converter.target_spec.supported_ops = [\n",
" tf.lite.OpsSet.TFLITE_BUILTINS, # enable TensorFlow Lite ops.\n",
" tf.lite.OpsSet.SELECT_TF_OPS # enable TensorFlow ops.\n",
"]\n",
"tflite_model = converter.convert()\n",
"\n",
"# Print the signatures from the converted model\n",
"interpreter = tf.lite.Interpreter(model_content=tflite_model)\n",
"signatures = interpreter.get_signature_list()\n",
"print(signatures)"
]
},
{
"cell_type": "markdown",
"id": "b5e85934",
"metadata": {
"id": "b5e85934"
},
"source": [
"## Run Signatures\n",
"\n",
"TensorFlow inference APIs support the signature-based executions:\n",
"\n",
"* Accessing the input/output tensors through the names of the inputs and\n",
" outputs, specified by the signature.\n",
"* Running each entry point of the graph separately, identified by the\n",
" signature key.\n",
"* Support for the SavedModel's initialization procedure.\n",
"\n",
"Java, C++ and Python language bindings are currently available. See example the\n",
"below sections.\n"
]
},
{
"cell_type": "markdown",
"id": "ZRBMFciMQmiB",
"metadata": {
"id": "ZRBMFciMQmiB"
},
"source": [
"\n",
"### Java"
]
},
{
"cell_type": "markdown",
"id": "04c5a4fc",
"metadata": {
"id": "04c5a4fc"
},
"source": [
"```\n",
"try (Interpreter interpreter = new Interpreter(file_of_tensorflowlite_model)) {\n",
" // Run encoding signature.\n",
" Map<String, Object> inputs = new HashMap<>();\n",
" inputs.put(\"x\", input);\n",
" Map<String, Object> outputs = new HashMap<>();\n",
" outputs.put(\"encoded_result\", encoded_result);\n",
" interpreter.runSignature(inputs, outputs, \"encode\");\n",
"\n",
" // Run decoding signature.\n",
" Map<String, Object> inputs = new HashMap<>();\n",
" inputs.put(\"x\", encoded_result);\n",
" Map<String, Object> outputs = new HashMap<>();\n",
" outputs.put(\"decoded_result\", decoded_result);\n",
" interpreter.runSignature(inputs, outputs, \"decode\");\n",
"}\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "5ba86c64",
"metadata": {
"id": "5ba86c64"
},
"source": [
"### C++"
]
},
{
"cell_type": "markdown",
"id": "397ad6fd",
"metadata": {
"id": "397ad6fd"
},
"source": [
"```\n",
"SignatureRunner* encode_runner =\n",
" interpreter->GetSignatureRunner(\"encode\");\n",
"encode_runner->ResizeInputTensor(\"x\", {100});\n",
"encode_runner->AllocateTensors();\n",
"\n",
"TfLiteTensor* input_tensor = encode_runner->input_tensor(\"x\");\n",
"float* input = GetTensorData<float>(input_tensor);\n",
"// Fill `input`.\n",
"\n",
"encode_runner->Invoke();\n",
"\n",
"const TfLiteTensor* output_tensor = encode_runner->output_tensor(\n",
" \"encoded_result\");\n",
"float* output = GetTensorData<float>(output_tensor);\n",
"// Access `output`.\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "0f4c6ad4",
"metadata": {
"id": "0f4c6ad4"
},
"source": [
"### Python"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ab7b1963",
"metadata": {
"id": "ab7b1963"
},
"outputs": [],
"source": [
"# Load the TFLite model in TFLite Interpreter\n",
"interpreter = tf.lite.Interpreter(model_content=tflite_model)\n",
"\n",
"# Print the signatures from the converted model\n",
"signatures = interpreter.get_signature_list()\n",
"print('Signature:', signatures)\n",
"\n",
"# encode and decode are callable with input as arguments.\n",
"encode = interpreter.get_signature_runner('encode')\n",
"decode = interpreter.get_signature_runner('decode')\n",
"\n",
"# 'encoded' and 'decoded' are dictionaries with all outputs from the inference.\n",
"input = tf.constant([1, 2, 3], dtype=tf.float32)\n",
"print('Input:', input)\n",
"encoded = encode(x=input)\n",
"print('Encoded result:', encoded)\n",
"decoded = decode(x=encoded['encoded_result'])\n",
"print('Decoded result:', decoded)"
]
},
{
"cell_type": "markdown",
"id": "81b42e5b",
"metadata": {
"id": "81b42e5b"
},
"source": [
"## Known limitations\n",
"\n",
"* As TFLite interpreter does not gurantee thread safety, the signature runners\n",
" from the same interpreter won't be executed concurrently.\n",
"* Support for iOS/Swift is not available yet.\n",
"\n"
]
},
{
"cell_type": "markdown",
"id": "3032Iof6QqmJ",
"metadata": {
"id": "3032Iof6QqmJ"
},
"source": [
"## Updates\n",
"\n",
"* Version 2.7\n",
" - The multiple signature feature is implemented.\n",
" - All the converter APIs from version two generate signature-enabled\n",
" TensorFlow Lite models.\n",
"* Version 2.5\n",
" - Signature feature is available through the `from_saved_model` converter\n",
" API."
]
}
],
"metadata": {
"colab": {
"collapsed_sections": [],
"name": "Signatures in TensorFlow Lite",
"provenance": [],
"toc_visible": true
},
"id": "a1b42e5b",
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
},
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 5
}