wehub/tensorflow--tensorflow
1
0
Fork 0
Code Issues Pull Requests Actions 6 Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution
cffconvert / validate (push) Has been skipped
Details
License Check / license-check (push) Failing after 2s
Details

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 12:14:16 +08:00
commit 8a852e4b4e
36502 changed files with 9277225 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tensorflow/lite/g3doc/models/convert/api_updates.md
+48
View File
@@ -0,0 +1,48 @@
# API Updates <a name="api_updates"></a>
This page provides information about updates made to the
`tf.lite.TFLiteConverter` [Python API](index.md) in TensorFlow 2.x.
Note: If any of the changes raise concerns, please file a
[GitHub issue](https://github.com/tensorflow/tensorflow/issues/new?template=60-tflite-converter-issue.md).
* TensorFlow 2.3
* Support integer (previously, only float) input/output type for integer
quantized models using the new `inference_input_type` and
`inference_output_type` attributes. Refer to this
[example usage](../../performance/post_training_quantization.md#integer_only).
* Support conversion and resizing of models with dynamic dimensions.
* Added a new experimental quantization mode with 16-bit activations and
8-bit weights.
* TensorFlow 2.2
* By default, leverage [MLIR-based conversion](https://mlir.llvm.org/),
Google's cutting edge compiler technology for machine learning. This
enables conversion of new classes of models, including Mask R-CNN,
Mobile BERT, etc and supports models with functional control flow.
* TensorFlow 2.0 vs TensorFlow 1.x
* Renamed the `target_ops` attribute to `target_spec.supported_ops`
* Removed the following attributes:
* _quantization_: `inference_type`, `quantized_input_stats`,
`post_training_quantize`, `default_ranges_stats`,
`reorder_across_fake_quant`, `change_concat_input_ranges`,
`get_input_arrays()`. Instead,
[quantize aware training](https://www.tensorflow.org/model_optimization/guide/quantization/training)
is supported through the `tf.keras` API and
[post training quantization](../../performance/post_training_quantization.md)
uses fewer attributes.
* _visualization_: `output_format`, `dump_graphviz_dir`,
`dump_graphviz_video`. Instead, the recommended approach for
visualizing a TensorFlow Lite model is to use
[visualize.py](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/tools/visualize.py).
* _frozen graphs_: `drop_control_dependency`, as frozen graphs are
unsupported in TensorFlow 2.x.
* Removed other converter APIs such as `tf.lite.toco_convert` and
`tf.lite.TocoConverter`
* Removed other related APIs such as `tf.lite.OpHint` and
`tf.lite.constants` (the `tf.lite.constants.*` types have been mapped to
`tf.*` TensorFlow data types, to reduce duplication)
tensorflow/lite/g3doc/models/convert/convert_models.md
+231
View File
@@ -0,0 +1,231 @@
# Convert TensorFlow models
This page describes how to convert a TensorFlow model
to a TensorFlow Lite model (an optimized
[FlatBuffer](https://google.github.io/flatbuffers/) format identified by the
`.tflite` file extension) using the TensorFlow Lite converter.
Note: This guide assumes you've both
[installed TensorFlow 2.x](https://www.tensorflow.org/install/pip#tensorflow-2-packages-are-available)
and trained models in TensorFlow 2.x.
If your model is trained in TensorFlow 1.x, considering
[migrating to TensorFlow 2.x](https://www.tensorflow.org/guide/migrate/tflite).
To identify the installed TensorFlow version, run
`print(tf.__version__)`.
## Conversion workflow
The diagram below illustrations the high-level workflow for converting
your model:
![TFLite converter workflow](../../images/convert/convert.png)
**Figure 1.** Converter workflow.
You can convert your model using one of the following options:
1. [Python API](#python_api) (***recommended***):
This allows you to integrate the conversion into your development pipeline,
apply optimizations, add metadata and many other tasks that simplify
the conversion process.
2. [Command line](#cmdline): This only supports basic model conversion.
Note: In case you encounter any issues during model conversion, create a
[GitHub issue](https://github.com/tensorflow/tensorflow/issues/new?template=60-tflite-converter-issue.md).
## Python API <a name="python_api"></a>
*Helper code: To learn more about the TensorFlow Lite converter
API, run `print(help(tf.lite.TFLiteConverter))`.*
Convert a TensorFlow model using
[`tf.lite.TFLiteConverter`](https://www.tensorflow.org/api_docs/python/tf/lite/TFLiteConverter).
A TensorFlow model is stored using the SavedModel format and is
generated either using the high-level `tf.keras.*` APIs (a Keras model) or
the low-level `tf.*` APIs (from which you generate concrete functions). As a
result, you have the following three options (examples are in the next few
sections):
* `tf.lite.TFLiteConverter.from_saved_model()` (**recommended**): Converts
a [SavedModel](https://www.tensorflow.org/guide/saved_model).
* `tf.lite.TFLiteConverter.from_keras_model()`: Converts a
[Keras](https://www.tensorflow.org/guide/keras/overview) model.
* `tf.lite.TFLiteConverter.from_concrete_functions()`: Converts
[concrete functions](https://www.tensorflow.org/guide/intro_to_graphs).
### Convert a SavedModel (recommended) <a name="saved_model"></a>
The following example shows how to convert a
[SavedModel](https://www.tensorflow.org/guide/saved_model) into a TensorFlow
Lite model.
```python
import tensorflow as tf
# Convert the model
converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir) # path to the SavedModel directory
tflite_model = converter.convert()
# Save the model.
with open('model.tflite', 'wb') as f:
f.write(tflite_model)
```
### Convert a Keras model <a name="keras"></a>
The following example shows how to convert a
[Keras](https://www.tensorflow.org/guide/keras/overview) model into a TensorFlow
Lite model.
```python
import tensorflow as tf
# Create a model using high-level tf.keras.* APIs
model = tf.keras.models.Sequential([
tf.keras.layers.Dense(units=1, input_shape=[1]),
tf.keras.layers.Dense(units=16, activation='relu'),
tf.keras.layers.Dense(units=1)
])
model.compile(optimizer='sgd', loss='mean_squared_error') # compile the model
model.fit(x=[-1, 0, 1], y=[-3, -1, 1], epochs=5) # train the model
# (to generate a SavedModel) tf.saved_model.save(model, "saved_model_keras_dir")
# Convert the model.
converter = tf.lite.TFLiteConverter.from_keras_model(model)
tflite_model = converter.convert()
# Save the model.
with open('model.tflite', 'wb') as f:
f.write(tflite_model)
```
### Convert concrete functions <a name="concrete_function"></a>
The following example shows how to convert
[concrete functions](https://www.tensorflow.org/guide/intro_to_graphs) into a
TensorFlow Lite model.
```python
import tensorflow as tf
# Create a model using low-level tf.* APIs
class Squared(tf.Module):
@tf.function(input_signature=[tf.TensorSpec(shape=[None], dtype=tf.float32)])
def __call__(self, x):
return tf.square(x)
model = Squared()
# (ro run your model) result = Squared(5.0) # This prints "25.0"
# (to generate a SavedModel) tf.saved_model.save(model, "saved_model_tf_dir")
concrete_func = model.__call__.get_concrete_function()
# Convert the model.
converter = tf.lite.TFLiteConverter.from_concrete_functions([concrete_func],
model)
tflite_model = converter.convert()
# Save the model.
with open('model.tflite', 'wb') as f:
f.write(tflite_model)
```
### Other features
* Apply [optimizations](../../performance/model_optimization.md). A common
optimization used is
[post training quantization](../../performance/post_training_quantization.md),
which can further reduce your model latency and size with minimal loss in
accuracy.
* Add [metadata](metadata.md), which makes it easier to create platform
specific wrapper code when deploying models on devices.
### Conversion errors
The following are common conversion errors and their solutions:
* Error: `Some ops are not supported by the native TFLite runtime, you can
enable TF kernels fallback using TF Select. See instructions:
https://www.tensorflow.org/lite/guide/ops_select. TF Select ops: ..., ..,
...`
Solution: The error occurs as your model has TF ops that don't have a
corresponding TFLite implementation. You can resolve this by
[using the TF op in the TFLite model](../../guide/ops_select.md)
(recommended). If you want to generate a model with TFLite ops only, you can
either add a request for the missing TFLite op in
[Github issue #21526](https://github.com/tensorflow/tensorflow/issues/21526)
(leave a comment if your request hasn’t already been mentioned) or
[create the TFLite op](../../guide/ops_custom.md#create_and_register_the_operator)
yourself.
* Error: `.. is neither a custom op nor a flex op`
Solution: If this TF op is:
* Supported in TF: The error occurs because the TF op is missing from the
[allowlist](../../guide/op_select_allowlist.md) (an exhaustive list of
TF ops supported by TFLite). You can resolve this as follows:
1. [Add missing ops to the allowlist](../../guide/op_select_allowlist.md#add_tensorflow_core_operators_to_the_allowed_list).
2. [Convert the TF model to a TFLite model and run inference](../../guide/ops_select.md).
* Unsupported in TF: The error occurs because TFLite is unaware of the
custom TF operator defined by you. You can resolve this as follows:
1. [Create the TF op](https://www.tensorflow.org/guide/create_op).
2. [Convert the TF model to a TFLite model](../../guide/op_select_allowlist.md#users_defined_operators).
3. [Create the TFLite op](../../guide/ops_custom.md#create_and_register_the_operator)
and run inference by linking it to the TFLite runtime.
## Command Line Tool <a name="cmdline"></a>
**Note:** It is highly recommended that you use the [Python API](#python_api)
listed above instead, if possible.
If you've
[installed TensorFlow 2.x from pip](https://www.tensorflow.org/install/pip), use
the `tflite_convert` command. To view all the available flags, use the
following command:
```sh
$ tflite_convert --help
`--output_file`. Type: string. Full path of the output file.
`--saved_model_dir`. Type: string. Full path to the SavedModel directory.
`--keras_model_file`. Type: string. Full path to the Keras H5 model file.
`--enable_v1_converter`. Type: bool. (default False) Enables the converter and flags used in TF 1.x instead of TF 2.x.
You are required to provide the `--output_file` flag and either the `--saved_model_dir` or `--keras_model_file` flag.
```
If you have the
[TensorFlow 2.x source](https://www.tensorflow.org/install/source)
downloaded and want to run the converter from that source without building and
installing the package,
you can replace '`tflite_convert`' with
'`bazel run tensorflow/lite/python:tflite_convert --`' in the command.
### Converting a SavedModel <a name="cmdline_saved_model"></a>
```sh
tflite_convert \
--saved_model_dir=/tmp/mobilenet_saved_model \
--output_file=/tmp/mobilenet.tflite
```
### Converting a Keras H5 model <a name="cmdline_keras_model"></a>
```sh
tflite_convert \
--keras_model_file=/tmp/mobilenet_keras_model.h5 \
--output_file=/tmp/mobilenet.tflite
```
## Next Steps
Use the [TensorFlow Lite interpreter](../../guide/inference.md) to run inference
on a client device (e.g. mobile, embedded).
tensorflow/lite/g3doc/models/convert/index.md
+150
View File
@@ -0,0 +1,150 @@
# Model conversion overview
The machine learning (ML) models you use with TensorFlow Lite are originally
built and trained using TensorFlow core libraries and tools. Once you've built
a model with TensorFlow core, you can convert it to a smaller, more
efficient ML model format called a TensorFlow Lite model.
This section provides guidance for converting
your TensorFlow models to the TensorFlow Lite model format.
Note: If you don't have a model to convert yet, see the
[Models overview](../)
page for guidance on choosing or building models.
## Conversion workflow
Converting TensorFlow models to TensorFlow Lite format can take a few paths
depending on the content of your ML model. As the first step of that process,
you should evaluate your model to determine if it can be directly converted.
This evaluation determines if the content of the model is supported by the
standard TensorFlow Lite runtime environments based on the TensorFlow operations
it uses. If your model uses operations outside of the supported set, you have
the option to refactor your model or use advanced conversion techniques.
The diagram below shows the high level steps in converting a model.
![TFLite conversion workflow](../../images/convert/convert_workflow_diag.png)
**Figure 1.** TensorFlow Lite conversion workflow.
The following sections outline the process of evaluating and converting models
for use with TensorFlow Lite.
### Input model formats
You can use the converter with the following input model formats:
* [SavedModel](https://www.tensorflow.org/guide/saved_model)
(***recommended***): A TensorFlow model saved as a set of files on disk.
* [Keras model](https://www.tensorflow.org/guide/keras/overview):
A model created using the high level Keras API.
* [Keras H5 format](https://www.tensorflow.org/guide/keras/save_and_serialize#keras_h5_format):
A light-weight alternative to SavedModel format supported by Keras API.
* [Models built from concrete functions](https://www.tensorflow.org/guide/intro_to_graphs):
A model created using the low level TensorFlow API.
You can save both the Keras and concrete function models as a SavedModel
and convert using the recommeded path.
Note: To avoid errors during inference, include signatures when exporting to the
SavedModel format.
The TensorFlow converter supports converting TensorFlow model's
input/output specifications to TensorFlow Lite models. See the topic
on [adding signatures](https://tensorflow.org/lite/guide/signatures).
If you have a Jax model, you can use the `TFLiteConverter.experimental_from_jax`
API to convert it to the TensorFlow Lite format. Note that this API is subject
to change while in experimental mode.
### Conversion evaluation
Evaluating your model is an important step before attempting to convert it.
When evaluating,
you want to determine if the contents of your model is compatible with the
TensorFlow Lite format. You should also determine if your model is a good fit
for use on mobile and edge devices in terms of the size of data the model uses,
its hardware processing requirements, and the model's overall size and
complexity.
For many models, the converter should work out of the box. However,
TensorFlow Lite builtin operator library supports a subset of
TensorFlow core operators, which means some models may need additional
steps before converting to TensorFlow Lite.
Additionally some operations that are supported by TensorFlow Lite have
restricted usage requirements for performance reasons. See the
[operator compatibility](../../guide/ops_compatibility) guide
to determine if your model needs to be refactored for conversion.
Key Point: Most models can be directly converted to TensorFlow Lite format. Some
models may require refactoring or use of advanced conversion techniques to
make them compatible.
### Model conversion
The TensorFlow Lite converter takes a TensorFlow model and generates a
TensorFlow Lite model (an optimized
[FlatBuffer](https://google.github.io/flatbuffers/) format identified by the
`.tflite` file extension). You can load
a SavedModel or directly convert a model you create in code.
The converter takes 3 main flags (or options) that customize the conversion
for your model:
1. [Compatibility flags](../../guide/ops_compatibility) allow you to specify
whether the conversion should allow custom operators.
1. [Optimization flags](../../performance/model_optimization) allow you to
specify the type of optimization to apply
during conversion. The most commonly used optimization technique is
[post-training quantization]().
1. [Metadata flags](metadata) allow you to add metadata to the converted model
which makes it easier to create platform specific wrapper code when deploying
models on devices.
You can convert your model using the [Python API](convert_models#python_api) or
the [Command line](convert_models#cmdline) tool. See the
[Convert TF model](convert_models) guide for step by step
instructions on running the converter on your model.
Typically you would convert your model for the standard TensorFlow Lite
[runtime environment](../../android#runtime) or the
[Google Play services runtime environment](../../android/play_services)
for TensorFlow Lite (Beta). Some advanced use cases require
customization of model runtime environment, which require additional steps in
the conversion proceess. See the
[advanced runtime environment](../../android#adv_runtime) section of the Android
overview for more guidance.
## Advanced conversion
If you run into [errors](convert_models#conversion_errors)
while running the converter on your model, it's most likely that you have an
operator compatibility issue. Not all TensorFlow operations are
supported by TensorFlow
Lite. You can work around these issues by refactoring your model, or by using
advanced conversion options that allow you to create a modified TensorFlow Lite
format model and a custom runtime environment for that model.
* See the [Model compatibility overview](../../guide/ops_compatibility)
for more information on TensorFlow and TensorFlow Lite model compatibility
considerations.
* Topics under the Model compatibility overview cover advanced techniques for
refactoring your model, such as the [Select operators](../../guide/ops_select)
guide.
* For full list of operations and limitations see
[TensorFlow Lite Ops page](https://www.tensorflow.org/mlir/tfl_ops).
## Next steps
* See the [convert TF models](convert_models) guide to quickly get started on
converting your model.
* See the [optimization overview](../../performance/model_optimization) for
guidance on how to optimize your converted model using techniques like
[post-training quantization](../../performance/post_training_quantization).
* See the [Adding metadata overview](metadata) to learn how to add metadata to
your models. Metadata provides other uses a description of your model as well
as information that can be leveraged by code generators.
tensorflow/lite/g3doc/models/convert/metadata.md
+546
View File
@@ -0,0 +1,546 @@
# Adding metadata to TensorFlow Lite models
TensorFlow Lite metadata provides a standard for model descriptions. The
metadata is an important source of knowledge about what the model does and its
input / output information. The metadata consists of both
* human readable parts which convey the best practice when using the model,
and
* machine readable parts that can be leveraged by code generators, such as the
[TensorFlow Lite Android code generator](../../inference_with_metadata/codegen.md#generate-model-interfaces-with-tensorflow-lite-code-generator-codegen)
and the
[Android Studio ML Binding feature](../../inference_with_metadata/codegen.md#use-android-studio-ml-model-binding-mlbinding).
All image models published on
[TensorFlow Hub](https://tfhub.dev/s?deployment-format=lite) have been populated
with metadata.
## Model with metadata format
<center><img src="../../images/convert/model_with_metadata.png" alt="model_with_metadata" width="70%"></center>
<center>Figure 1. TFLite model with metadata and associated files.</center>
Model metadata is defined in
[metadata_schema.fbs](https://github.com/tensorflow/tflite-support/blob/master/tensorflow_lite_support/metadata/metadata_schema.fbs),
a
[FlatBuffer](https://google.github.io/flatbuffers/index.html#flatbuffers_overview)
file. As shown in Figure 1, it is stored in the
[metadata](https://github.com/tensorflow/tensorflow/blob/bd73701871af75539dd2f6d7fdba5660a8298caf/tensorflow/lite/schema/schema.fbs#L1208)
field of the
[TFLite model schema](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/schema/schema.fbs),
under the name, `"TFLITE_METADATA"`. Some models may come with associated files,
such as
[classification label files](https://github.com/tensorflow/examples/blob/dd98bc2b595157c03ac9fa47ac8659bb20aa8bbd/lite/examples/image_classification/android/models/src/main/assets/labels.txt#L1).
These files are concatenated to the end of the original model file as a ZIP
using the ZipFile
["append" mode](https://pymotw.com/2/zipfile/#appending-to-files) (`'a'` mode).
TFLite Interpreter can consume the new file format in the same way as before.
See [Pack the associated files](#pack-the-associated-files) for more
information.
See the instruction below about how to populate, visualize, and read metadata.
## Setup the metadata tools
Before adding metadata to your model, you will need to a Python programming
environment setup for running TensorFlow. There is a detailed guide on how to
set this up [here](https://www.tensorflow.org/install).
After setup the Python programming environment, you will need to install
additional tooling:
```sh
pip install tflite-support
```
TensorFlow Lite metadata tooling supports Python 3.
## Adding metadata using Flatbuffers Python API
Note: to create metadata for the popular ML tasks supported in
[TensorFlow Lite Task Library](https://ai.google.dev/edge/litert/libraries/task_library/overview),
use the high-level API in the
[TensorFlow Lite Metadata Writer Library](metadata_writer_tutorial.ipynb).
There are three parts to the model metadata in the
[schema](https://github.com/tensorflow/tflite-support/blob/master/tensorflow_lite_support/metadata/metadata_schema.fbs):
1. **Model information** - Overall description of the model as well as items
such as license terms. See
[ModelMetadata](https://github.com/tensorflow/tflite-support/blob/4cd0551658b6e26030e0ba7fc4d3127152e0d4ae/tensorflow_lite_support/metadata/metadata_schema.fbs#L640).
2. **Input information** - Description of the inputs and pre-processing
required such as normalization. See
[SubGraphMetadata.input_tensor_metadata](https://github.com/tensorflow/tflite-support/blob/4cd0551658b6e26030e0ba7fc4d3127152e0d4ae/tensorflow_lite_support/metadata/metadata_schema.fbs#L590).
3. **Output information** - Description of the output and post-processing
required such as mapping to labels. See
[SubGraphMetadata.output_tensor_metadata](https://github.com/tensorflow/tflite-support/blob/4cd0551658b6e26030e0ba7fc4d3127152e0d4ae/tensorflow_lite_support/metadata/metadata_schema.fbs#L599).
Since TensorFlow Lite only supports single subgraph at this point, the
[TensorFlow Lite code generator](../../inference_with_metadata/codegen..md#generate-model-interfaces-with-tensorflow-lite-code-generator-codegen)
and the
[Android Studio ML Binding feature](../../inference_with_metadata/codegen.md#use-android-studio-ml-model-binding-mlbinding)
will use `ModelMetadata.name` and `ModelMetadata.description`, instead of
`SubGraphMetadata.name` and `SubGraphMetadata.description`, when displaying
metadata and generating code.
### Supported Input / Output types
TensorFlow Lite metadata for input and output are not designed with specific
model types in mind but rather input and output types. It does not matter what
the model functionally does, as long as the input and output types consists of
the following or a combination of the following, it is supported by TensorFlow
Lite metadata:
* Feature - Numbers which are unsigned integers or float32.
* Image - Metadata currently supports RGB and greyscale images.
* Bounding box - Rectangular shape bounding boxes. The schema supports
[a variety of numbering schemes](https://github.com/tensorflow/tflite-support/blob/4cd0551658b6e26030e0ba7fc4d3127152e0d4ae/tensorflow_lite_support/metadata/metadata_schema.fbs#L214).
### Pack the associated files
TensorFlow Lite models may come with different associated files. For example,
natural language models usually have vocab files that map word pieces to word
IDs; classification models may have label files that indicate object categories.
Without the associated files (if there are), a model will not function well.
The associated files can now be bundled with the model through the metadata
Python library. The new TensorFlow Lite model becomes a zip file that contains
both the model and the associated files. It can be unpacked with common zip
tools. This new model format keeps using the same file extension, `.tflite`. It
is compatible with existing TFLite framework and Interpreter. See
[Pack metadata and associated files into the model](#pack-metadata-and-associated-files-into-the-model)
for more details.
The associated file information can be recorded in the metadata. Depending on
the file type and where the file is attached to (i.e. `ModelMetadata`,
`SubGraphMetadata`, and `TensorMetadata`),
[the TensorFlow Lite Android code generator](../../inference_with_metadata/codegen.md)
may apply corresponding pre/post processing automatically to the object. See
[the \<Codegen usage\> section of each associate file type](https://github.com/tensorflow/tflite-support/blob/4cd0551658b6e26030e0ba7fc4d3127152e0d4ae/tensorflow_lite_support/metadata/metadata_schema.fbs#L77-L127)
in the schema for more details.
### Normalization and quantization parameters
Normalization is a common data preprocessing technique in machine learning. The
goal of normalization is to change the values to a common scale, without
distorting differences in the ranges of values.
[Model quantization](https://www.tensorflow.org/lite/performance/model_optimization#model_quantization)
is a technique that allows for reduced precision representations of weights and
optionally, activations for both storage and computation.
In terms of preprocessing and post-processing, normalization and quantization
are two independent steps. Here are the details.
| | Normalization | Quantization |
| :---------------------: | ----------------------- | ------------------------ |
| \ | **Float model**: \ | **Float model**: \ |
: An example of the : - mean\: 127.5 \ : - zeroPoint\: 0 \ :
: parameter values of the : - std\: 127.5 \ : - scale\: 1.0 \ :
: input image in : **Quant model**\: \ : **Quant model**\: \ :
: MobileNet for float and : - mean\: 127.5 \ : - zeroPoint\: 128.0 \ :
: quant models, : - std\: 127.5 : - scale\:0.0078125f \ :
: respectively. : : :
| \ | \ | **Float models** does |
: \ : \ : not need quantization. \ :
: \ : **Inputs**\: If input : **Quantized model** may :
: \ : data is normalized in : or may not need :
: When to invoke? : training, the input : quantization in pre/post :
: : data of inference needs : processing. It depends :
: : to be normalized : on the datatype of :
: : accordingly. \ : input/output tensors. \ :
: : **Outputs**\: output : - float tensors\: no :
: : data will not be : quantization in pre/post :
: : normalized in general. : processing needed. Quant :
: : : op and dequant op are :
: : : baked into the model :
: : : graph. \ :
: : : - int8/uint8 tensors\: :
: : : need quantization in :
: : : pre/post processing. :
| \ | \ | **Quantize for inputs**: |
: \ : \ : \ :
: Formula : normalized_input = : q = f / scale + :
: : (input - mean) / std : zeroPoint \ :
: : : **Dequantize for :
: : : outputs**\: \ :
: : : f = (q - zeroPoint) * :
: : : scale :
| \ | Filled by model creator | Filled automatically by |
: Where are the : and stored in model : TFLite converter, and :
: parameters : metadata, as : stored in tflite model :
: : `NormalizationOptions` : file. :
| How to get the | Through the | Through the TFLite |
: parameters? : `MetadataExtractor` API : `Tensor` API [1] or :
: : [2] : through the :
: : : `MetadataExtractor` API :
: : : [2] :
| Do float and quant | Yes, float and quant | No, the float model does |
: models share the same : models have the same : not need quantization. :
: value? : Normalization : :
: : parameters : :
| Does TFLite Code | \ | \ |
: generator or Android : Yes : Yes :
: Studio ML binding : : :
: automatically generate : : :
: it in data processing? : : :
[1] The
[TensorFlow Lite Java API](https://github.com/tensorflow/tensorflow/blob/09ec15539eece57b257ce9074918282d88523d56/tensorflow/lite/java/src/main/java/org/tensorflow/lite/Tensor.java#L73)
and the
[TensorFlow Lite C++ API](https://github.com/tensorflow/tensorflow/blob/09ec15539eece57b257ce9074918282d88523d56/tensorflow/lite/c/common.h#L391).
\
[2] The [metadata extractor library](#read-the-metadata-from-models)
When processing image data for uint8 models, normalization and quantization are
sometimes skipped. It is fine to do so when the pixel values are in the range of
[0, 255]. But in general, you should always process the data according to the
normalization and quantization parameters when applicable.
[TensorFlow Lite Task Library](https://www.tensorflow.org/lite/inference_with_metadata/overview)
can handle normalization for you if you set up `NormalizationOptions` in
metadata. Quantization and dequantization processing is always encapsulated.
### Examples
Note: The export directory specified has to exist before you run the script; it
does not get created as part of the process.
You can find examples on how the metadata should be populated for different
types of models here:
#### Image classification
Download the script
[here](https://github.com/tensorflow/examples/tree/master/lite/examples/image_classification/metadata/metadata_writer_for_image_classifier.py)
, which populates metadata to
[mobilenet_v1_0.75_160_quantized.tflite](https://tfhub.dev/tensorflow/lite-model/mobilenet_v1_0.75_160_quantized/1/default/1).
Run the script like this:
```sh
python ./metadata_writer_for_image_classifier.py \
--model_file=./model_without_metadata/mobilenet_v1_0.75_160_quantized.tflite \
--label_file=./model_without_metadata/labels.txt \
--export_directory=model_with_metadata
```
To populate metadata for other image classification models, add the model specs
like
[this](https://github.com/tensorflow/examples/blob/master/lite/examples/image_classification/metadata/metadata_writer_for_image_classifier.py#L63-L74)
into the script. The rest of this guide will highlight some of the key sections
in the image classification example to illustrate the key elements.
### Deep dive into the image classification example
#### Model information
Metadata starts by creating a new model info:
```python
from tflite_support import flatbuffers
from tflite_support import metadata as _metadata
from tflite_support import metadata_schema_py_generated as _metadata_fb
""" ... """
"""Creates the metadata for an image classifier."""
# Creates model info.
model_meta = _metadata_fb.ModelMetadataT()
model_meta.name = "MobileNetV1 image classifier"
model_meta.description = ("Identify the most prominent object in the "
"image from a set of 1,001 categories such as "
"trees, animals, food, vehicles, person etc.")
model_meta.version = "v1"
model_meta.author = "TensorFlow"
model_meta.license = ("Apache License. Version 2.0 "
"http://www.apache.org/licenses/LICENSE-2.0.")
```
#### Input / output information
This section shows you how to describe your model's input and output signature.
This metadata may be used by automatic code generators to create pre- and post-
processing code. To create input or output information about a tensor:
```python
# Creates input info.
input_meta = _metadata_fb.TensorMetadataT()
# Creates output info.
output_meta = _metadata_fb.TensorMetadataT()
```
#### Image input
Image is a common input type for machine learning. TensorFlow Lite metadata
supports information such as colorspace and pre-processing information such as
normalization. The dimension of the image does not require manual specification
since it is already provided by the shape of the input tensor and can be
automatically inferred.
```python
input_meta.name = "image"
input_meta.description = (
"Input image to be classified. The expected image is {0} x {1}, with "
"three channels (red, blue, and green) per pixel. Each value in the "
"tensor is a single byte between 0 and 255.".format(160, 160))
input_meta.content = _metadata_fb.ContentT()
input_meta.content.contentProperties = _metadata_fb.ImagePropertiesT()
input_meta.content.contentProperties.colorSpace = (
_metadata_fb.ColorSpaceType.RGB)
input_meta.content.contentPropertiesType = (
_metadata_fb.ContentProperties.ImageProperties)
input_normalization = _metadata_fb.ProcessUnitT()
input_normalization.optionsType = (
_metadata_fb.ProcessUnitOptions.NormalizationOptions)
input_normalization.options = _metadata_fb.NormalizationOptionsT()
input_normalization.options.mean = [127.5]
input_normalization.options.std = [127.5]
input_meta.processUnits = [input_normalization]
input_stats = _metadata_fb.StatsT()
input_stats.max = [255]
input_stats.min = [0]
input_meta.stats = input_stats
```
#### Label output
Label can be mapped to an output tensor via an associated file using
`TENSOR_AXIS_LABELS`.
```python
# Creates output info.
output_meta = _metadata_fb.TensorMetadataT()
output_meta.name = "probability"
output_meta.description = "Probabilities of the 1001 labels respectively."
output_meta.content = _metadata_fb.ContentT()
output_meta.content.content_properties = _metadata_fb.FeaturePropertiesT()
output_meta.content.contentPropertiesType = (
_metadata_fb.ContentProperties.FeatureProperties)
output_stats = _metadata_fb.StatsT()
output_stats.max = [1.0]
output_stats.min = [0.0]
output_meta.stats = output_stats
label_file = _metadata_fb.AssociatedFileT()
label_file.name = os.path.basename("your_path_to_label_file")
label_file.description = "Labels for objects that the model can recognize."
label_file.type = _metadata_fb.AssociatedFileType.TENSOR_AXIS_LABELS
output_meta.associatedFiles = [label_file]
```
#### Create the metadata Flatbuffers
The following code combines the model information with the input and output
information:
```python
# Creates subgraph info.
subgraph = _metadata_fb.SubGraphMetadataT()
subgraph.inputTensorMetadata = [input_meta]
subgraph.outputTensorMetadata = [output_meta]
model_meta.subgraphMetadata = [subgraph]
b = flatbuffers.Builder(0)
b.Finish(
model_meta.Pack(b),
_metadata.MetadataPopulator.METADATA_FILE_IDENTIFIER)
metadata_buf = b.Output()
```
#### Pack metadata and associated files into the model
Once the metadata Flatbuffers is created, the metadata and the label file are
written into the TFLite file via the `populate` method:
```python
populator = _metadata.MetadataPopulator.with_model_file(model_file)
populator.load_metadata_buffer(metadata_buf)
populator.load_associated_files(["your_path_to_label_file"])
populator.populate()
```
You can pack as many associated files as you want into the model through
`load_associated_files`. However, it is required to pack at least those files
documented in the metadata. In this example, packing the label file is
mandatory.
## Visualize the metadata
You can use [Netron](https://github.com/lutzroeder/netron) to visualize your
metadata, or you can read the metadata from a TensorFlow Lite model into a json
format using the `MetadataDisplayer`:
```python
displayer = _metadata.MetadataDisplayer.with_model_file(export_model_path)
export_json_file = os.path.join(FLAGS.export_directory,
os.path.splitext(model_basename)[0] + ".json")
json_file = displayer.get_metadata_json()
# Optional: write out the metadata as a json file
with open(export_json_file, "w") as f:
f.write(json_file)
```
Android Studio also supports displaying metadata through the
[Android Studio ML Binding feature](https://developer.android.com/studio/preview/features#tensor-flow-lite-models).
## Metadata versioning
The
[metadata schema](https://github.com/tensorflow/tflite-support/blob/master/tensorflow_lite_support/metadata/metadata_schema.fbs)
is versioned both by the Semantic versioning number, which tracks the changes of
the schema file, and by the Flatbuffers file identification, which indicates the
true version compatibility.
### The Semantic versioning number
The metadata schema is versioned by the
[Semantic versioning number](https://github.com/tensorflow/tflite-support/blob/4cd0551658b6e26030e0ba7fc4d3127152e0d4ae/tensorflow_lite_support/metadata/metadata_schema.fbs#L53),
such as MAJOR.MINOR.PATCH. It tracks schema changes according to the rules
[here](https://github.com/tensorflow/tflite-support/blob/4cd0551658b6e26030e0ba7fc4d3127152e0d4ae/tensorflow_lite_support/metadata/metadata_schema.fbs#L32-L44).
See the
[history of fields](https://github.com/tensorflow/tflite-support/blob/4cd0551658b6e26030e0ba7fc4d3127152e0d4ae/tensorflow_lite_support/metadata/metadata_schema.fbs#L63)
added after version `1.0.0`.
### The Flatbuffers file identification
Semantic versioning guarantees the compatibility if following the rules, but it
does not imply the true incompatibility. When bumping up the MAJOR number, it
does not necessarily mean the backward compatibility is broken. Therefore, we
use the
[Flatbuffers file identification](https://google.github.io/flatbuffers/md__schemas.html),
[file_identifier](https://github.com/tensorflow/tflite-support/blob/4cd0551658b6e26030e0ba7fc4d3127152e0d4ae/tensorflow_lite_support/metadata/metadata_schema.fbs#L61),
to denote the true compatibility of the metadata schema. The file identifier is
exactly 4 characters long. It is fixed to a certain metadata schema and not
subject to change by users. If the backward compatibility of the metadata schema
has to be broken for some reason, the file_identifier will bump up, for example,
from “M001” to “M002”. File_identifier is expected to be changed much less
frequently than the metadata_version.
### The minimum necessary metadata parser version
The
[minimum necessary metadata parser version](https://github.com/tensorflow/tflite-support/blob/4cd0551658b6e26030e0ba7fc4d3127152e0d4ae/tensorflow_lite_support/metadata/metadata_schema.fbs#L681)
is the minimum version of metadata parser (the Flatbuffers generated code) that
can read the metadata Flatbuffers in full. The version is effectively the
largest version number among the versions of all the fields populated and the
smallest compatible version indicated by the file identifier. The minimum
necessary metadata parser version is automatically populated by the
`MetadataPopulator` when the metadata is populated into a TFLite model. See the
[metadata extractor](#read-the-metadata-from-models) for more information on how
the minimum necessary metadata parser version is used.
## Read the metadata from models
The Metadata Extractor library is convenient tool to read the metadata and
associated files from a models across different platforms (see the
[Java version](https://github.com/tensorflow/tflite-support/tree/master/tensorflow_lite_support/metadata/java)
and the
[C++ version](https://github.com/tensorflow/tflite-support/tree/master/tensorflow_lite_support/metadata/cc)).
You can build your own metadata extractor tool in other languages using the
Flatbuffers library.
### Read the metadata in Java
To use the Metadata Extractor library in your Android app, we recommend using
the
[TensorFlow Lite Metadata AAR hosted at MavenCentral](https://search.maven.org/artifact/org.tensorflow/tensorflow-lite-metadata).
It contains the `MetadataExtractor` class, as well as the FlatBuffers Java
bindings for the
[metadata schema](https://github.com/tensorflow/tflite-support/blob/master/tensorflow_lite_support/metadata/metadata_schema.fbs)
and the
[model schema](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/schema/schema.fbs).
You can specify this in your `build.gradle` dependencies as follows:
```build
dependencies {
implementation 'org.tensorflow:tensorflow-lite-metadata:0.1.0'
}
```
To use nightly snapshots, make sure that you have added
[Sonatype snapshot repository](https://www.tensorflow.org/lite/android/lite_build#use_nightly_snapshots).
You can initialize a `MetadataExtractor` object with a `ByteBuffer` that points
to the model:
```java
public MetadataExtractor(ByteBuffer buffer);
```
The `ByteBuffer` must remain unchanged for the entire lifetime of the
`MetadataExtractor` object. The initialization may fail if the Flatbuffers file
identifier of the model metadata does not match that of the metadata parser. See
[metadata versioning](#metadata-versioning) for more information.
With matching file identifiers, the metadata extractor will successfully read
metadata generated from all past and future schema due to the Flatbuffers'
forwards and backward compatibility mechanism. However, fields from future
schemas cannot be extracted by older metadata extractors. The
[minimum necessary parser version](#the-minimum-necessary-metadata-parser-version)
of the metadata indicates the minimum version of metadata parser that can read
the metadata Flatbuffers in full. You can use the following method to verify if
the minimum necessary parser version condition is met:
```java
public final boolean isMinimumParserVersionSatisfied();
```
Passing in a model without metadata is allowed. However, invoking methods that
read from the metadata will cause runtime errors. You can check if a model has
metadata by invoking the `hasMetadata` method:
```java
public boolean hasMetadata();
```
`MetadataExtractor` provides convenient functions for you to get the
input/output tensors' metadata. For example,
```java
public int getInputTensorCount();
public TensorMetadata getInputTensorMetadata(int inputIndex);
public QuantizationParams getInputTensorQuantizationParams(int inputIndex);
public int[] getInputTensorShape(int inputIndex);
public int getoutputTensorCount();
public TensorMetadata getoutputTensorMetadata(int inputIndex);
public QuantizationParams getoutputTensorQuantizationParams(int inputIndex);
public int[] getoutputTensorShape(int inputIndex);
```
Though the
[TensorFlow Lite model schema](https://github.com/tensorflow/tensorflow/blob/aa7ff6aa28977826e7acae379e82da22482b2bf2/tensorflow/lite/schema/schema.fbs#L1075)
supports multiple subgraphs, the TFLite Interpreter currently only supports a
single subgraph. Therefore, `MetadataExtractor` omits subgraph index as an input
argument in its methods.
## Read the associated files from models
The TensorFlow Lite model with metadata and associated files is essentially a
zip file that can be unpacked with common zip tools to get the associated files.
For example, you can unzip
[mobilenet_v1_0.75_160_quantized](https://tfhub.dev/tensorflow/lite-model/mobilenet_v1_0.75_160_quantized/1/metadata/1)
and extract the label file in the model as follows:
```sh
$ unzip mobilenet_v1_0.75_160_quantized_1_metadata_1.tflite
Archive: mobilenet_v1_0.75_160_quantized_1_metadata_1.tflite
extracting: labels.txt
```
You can also read associated files through the Metadata Extractor library.
In Java, pass the file name into the `MetadataExtractor.getAssociatedFile`
method:
```java
public InputStream getAssociatedFile(String fileName);
```
Similarly, in C++, this can be done with the method,
`ModelMetadataExtractor::GetAssociatedFile`:
```c++
tflite::support::StatusOr<absl::string_view> GetAssociatedFile(
const std::string& filename) const;
```
tensorflow/lite/g3doc/models/convert/metadata_writer_tutorial.ipynb
+847
View File
@@ -0,0 +1,847 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "Mq-riZs-TJGt"
},
"source": [
"##### Copyright 2024 The AI Edge Authors."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "LEvnopDoTC4M"
},
"outputs": [],
"source": [
"#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "QSRG6qmtTRSk"
},
"source": [
"# TensorFlow Lite Metadata Writer API\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JlzjEt4Txr0x"
},
"source": [
"<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
" <td>\n",
" <a target=\"_blank\" href=\"https://www.tensorflow.org/lite/models/convert/metadata_writer_tutorial\"><img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" />View on TensorFlow.org</a>\n",
" </td>\n",
" <td>\n",
" <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/models/convert/metadata_writer_tutorial.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
" </td>\n",
" <td>\n",
" <a target=\"_blank\" href=\"https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/models/convert/metadata_writer_tutorial.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://storage.googleapis.com/tensorflow_docs/tensorflow/tensorflow/lite/g3doc/models/convert/metadata_writer_tutorial.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\" />Download notebook</a>\n",
" </td>\n",
"\n",
"</table>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "b0gwEhfRYat6"
},
"source": [
"[TensorFlow Lite Model Metadata](https://www.tensorflow.org/lite/models/convert/metadata) is a standard model description format. It contains rich semantics for general model information, inputs/outputs, and associated files, which makes the model self-descriptive and exchangeable.\n",
"\n",
"Model Metadata is currently used in the following two primary use cases:\n",
"1. **Enable easy model inference using TensorFlow Lite [Task Library](https://www.tensorflow.org/lite/inference_with_metadata/task_library/overview) and [codegen tools](https://www.tensorflow.org/lite/inference_with_metadata/codegen)**. Model Metadata contains the mandatory information required during inference, such as label files in image classification, sampling rate of the audio input in audio classification, and tokenizer type to process input string in Natural Language models.\n",
"\n",
"2. **Enable model creators to include documentation**, such as the description of model inputs/outputs or how to use the model. Model users can view this documentation via visualization tools such as [Netron](https://netron.app/).\n",
"\n",
"TensorFlow Lite Metadata Writer API provides an easy-to-use API to create Model Metadata for popular ML tasks supported by the TFLite Task Library. This notebook shows examples of how the metadata should be populated for the following tasks below:\n",
"\n",
"* [Image classifiers](#image_classifiers)\n",
"* [Object detectors](#object_detectors)\n",
"* [Image segmenters](#image_segmenters)\n",
"* [Natural language classifiers](#nl_classifiers)\n",
"* [Audio classifiers](#audio_classifiers)\n",
"\n",
"Metadata writers for BERT natural language classifiers and BERT question answerers are coming soon.\n",
"\n",
"If you want to add metadata for use cases that are not supported, please use the [Flatbuffers Python API](https://www.tensorflow.org/lite/models/convert/metadata#adding_metadata). See the tutorials [here](https://www.tensorflow.org/lite/models/convert/metadata#adding_metadata).\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "GVRIGdA4T6tO"
},
"source": [
"## Prerequisites"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "bVTD2KSyotBK"
},
"source": [
"Install the TensorFlow Lite Support Pypi package."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "m-8xSrSvUg-6"
},
"outputs": [],
"source": [
"!pip install tflite-support-nightly"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "hyYS87Odpxef"
},
"source": [
"## Create Model Metadata for Task Library and Codegen"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "uLxv541TqTim"
},
"source": [
"<a name=image_classifiers></a>\n",
"### Image classifiers"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "s41TjCGlsyEF"
},
"source": [
"See the [image classifier model compatibility requirements](https://www.tensorflow.org/lite/inference_with_metadata/task_library/image_classifier#model_compatibility_requirements) for more details about the supported model format."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_KsPKmg8T9-8"
},
"source": [
"Step 1: Import the required packages."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "hhgNqEtWrwB3"
},
"outputs": [],
"source": [
"from tflite_support.metadata_writers import image_classifier\n",
"from tflite_support.metadata_writers import writer_utils"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "o9WBgiFdsiIQ"
},
"source": [
"Step 2: Download the example image classifier, [mobilenet_v2_1.0_224.tflite](https://github.com/tensorflow/tflite-support/raw/master/tensorflow_lite_support/metadata/python/tests/testdata/image_classifier/mobilenet_v2_1.0_224.tflite), and the [label file](https://github.com/tensorflow/tflite-support/raw/master/tensorflow_lite_support/metadata/python/tests/testdata/image_classifier/labels.txt)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "6WgSBbNet-Tt"
},
"outputs": [],
"source": [
"!curl -L https://github.com/tensorflow/tflite-support/raw/master/tensorflow_lite_support/metadata/python/tests/testdata/image_classifier/mobilenet_v2_1.0_224.tflite -o mobilenet_v2_1.0_224.tflite\n",
"!curl -L https://github.com/tensorflow/tflite-support/raw/master/tensorflow_lite_support/metadata/python/tests/testdata/image_classifier/labels.txt -o mobilenet_labels.txt"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ALtlz7woweHe"
},
"source": [
"Step 3: Create metadata writer and populate."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "_SMEBBt2r-W6"
},
"outputs": [],
"source": [
"ImageClassifierWriter = image_classifier.MetadataWriter\n",
"_MODEL_PATH = \"mobilenet_v2_1.0_224.tflite\"\n",
"# Task Library expects label files that are in the same format as the one below.\n",
"_LABEL_FILE = \"mobilenet_labels.txt\"\n",
"_SAVE_TO_PATH = \"mobilenet_v2_1.0_224_metadata.tflite\"\n",
"# Normalization parameters are required when reprocessing the image. It is\n",
"# optional if the image pixel values are in the range of [0, 255] and the input\n",
"# tensor is quantized to uint8. See the introduction for normalization and\n",
"# quantization parameters below for more details.\n",
"# https://www.tensorflow.org/lite/models/convert/metadata#normalization_and_quantization_parameters)\n",
"_INPUT_NORM_MEAN = 127.5\n",
"_INPUT_NORM_STD = 127.5\n",
"\n",
"# Create the metadata writer.\n",
"writer = ImageClassifierWriter.create_for_inference(\n",
" writer_utils.load_file(_MODEL_PATH), [_INPUT_NORM_MEAN], [_INPUT_NORM_STD],\n",
" [_LABEL_FILE])\n",
"\n",
"# Verify the metadata generated by the metadata writer.\n",
"print(writer.get_metadata_json())\n",
"\n",
"# Populate the metadata into the model.\n",
"writer_utils.save_file(writer.populate(), _SAVE_TO_PATH)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "GhhTDkr-uf0n"
},
"source": [
"<a name=object_detectors></a>\n",
"### Object detectors"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "EL9GssnTuf0n"
},
"source": [
"See the [object detector model compatibility requirements](https://www.tensorflow.org/lite/inference_with_metadata/task_library/object_detector#model_compatibility_requirements) for more details about the supported model format."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "r-HUTEtHuf0n"
},
"source": [
"Step 1: Import the required packages."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "2_NIROeouf0o"
},
"outputs": [],
"source": [
"from tflite_support.metadata_writers import object_detector\n",
"from tflite_support.metadata_writers import writer_utils"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "UM6jijiUuf0o"
},
"source": [
"Step 2: Download the example object detector, [ssd_mobilenet_v1.tflite](https://github.com/tensorflow/tflite-support/blob/master/tensorflow_lite_support/metadata/python/tests/testdata/object_detector/ssd_mobilenet_v1.tflite), and the [label file](https://github.com/tensorflow/tflite-support/blob/master/tensorflow_lite_support/metadata/python/tests/testdata/object_detector/labelmap.txt)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "4i_BBfGzuf0o"
},
"outputs": [],
"source": [
"!curl -L https://github.com/tensorflow/tflite-support/raw/master/tensorflow_lite_support/metadata/python/tests/testdata/object_detector/ssd_mobilenet_v1.tflite -o ssd_mobilenet_v1.tflite\n",
"!curl -L https://github.com/tensorflow/tflite-support/raw/master/tensorflow_lite_support/metadata/python/tests/testdata/object_detector/labelmap.txt -o ssd_mobilenet_labels.txt"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DG9T3eSDwsnd"
},
"source": [
"Step 3: Create metadata writer and populate."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "vMGGeJfCuf0p"
},
"outputs": [],
"source": [
"ObjectDetectorWriter = object_detector.MetadataWriter\n",
"_MODEL_PATH = \"ssd_mobilenet_v1.tflite\"\n",
"# Task Library expects label files that are in the same format as the one below.\n",
"_LABEL_FILE = \"ssd_mobilenet_labels.txt\"\n",
"_SAVE_TO_PATH = \"ssd_mobilenet_v1_metadata.tflite\"\n",
"# Normalization parameters are required when reprocessing the image. It is\n",
"# optional if the image pixel values are in the range of [0, 255] and the input\n",
"# tensor is quantized to uint8. See the introduction for normalization and\n",
"# quantization parameters below for more details.\n",
"# https://www.tensorflow.org/lite/models/convert/metadata#normalization_and_quantization_parameters)\n",
"_INPUT_NORM_MEAN = 127.5\n",
"_INPUT_NORM_STD = 127.5\n",
"\n",
"# Create the metadata writer.\n",
"writer = ObjectDetectorWriter.create_for_inference(\n",
" writer_utils.load_file(_MODEL_PATH), [_INPUT_NORM_MEAN], [_INPUT_NORM_STD],\n",
" [_LABEL_FILE])\n",
"\n",
"# Verify the metadata generated by the metadata writer.\n",
"print(writer.get_metadata_json())\n",
"\n",
"# Populate the metadata into the model.\n",
"writer_utils.save_file(writer.populate(), _SAVE_TO_PATH)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "QT0Oa0SU6uGS"
},
"source": [
"<a name=image_segmenters></a>\n",
"### Image segmenters"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "XaFQmg-S6uGW"
},
"source": [
"See the [image segmenter model compatibility requirements](https://www.tensorflow.org/lite/inference_with_metadata/task_library/image_segmenter#model_compatibility_requirements) for more details about the supported model format."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DiktANhj6uGX"
},
"source": [
"Step 1: Import the required packages."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "H6Lrw3op6uGX"
},
"outputs": [],
"source": [
"from tflite_support.metadata_writers import image_segmenter\n",
"from tflite_support.metadata_writers import writer_utils"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "9EFs8Oyi6uGX"
},
"source": [
"Step 2: Download the example image segmenter, [deeplabv3.tflite](https://github.com/tensorflow/tflite-support/blob/master/tensorflow_lite_support/metadata/python/tests/testdata/image_segmenter/deeplabv3.tflite), and the [label file](https://github.com/tensorflow/tflite-support/blob/master/tensorflow_lite_support/metadata/python/tests/testdata/image_segmenter/labelmap.txt)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "feQDH0bN6uGY"
},
"outputs": [],
"source": [
"!curl -L https://github.com/tensorflow/tflite-support/raw/master/tensorflow_lite_support/metadata/python/tests/testdata/image_segmenter/deeplabv3.tflite -o deeplabv3.tflite\n",
"!curl -L https://github.com/tensorflow/tflite-support/raw/master/tensorflow_lite_support/metadata/python/tests/testdata/image_segmenter/labelmap.txt -o deeplabv3_labels.txt"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "8LhiAbJM6uGY"
},
"source": [
"Step 3: Create metadata writer and populate."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "yot8xLI46uGY"
},
"outputs": [],
"source": [
"ImageSegmenterWriter = image_segmenter.MetadataWriter\n",
"_MODEL_PATH = \"deeplabv3.tflite\"\n",
"# Task Library expects label files that are in the same format as the one below.\n",
"_LABEL_FILE = \"deeplabv3_labels.txt\"\n",
"_SAVE_TO_PATH = \"deeplabv3_metadata.tflite\"\n",
"# Normalization parameters are required when reprocessing the image. It is\n",
"# optional if the image pixel values are in the range of [0, 255] and the input\n",
"# tensor is quantized to uint8. See the introduction for normalization and\n",
"# quantization parameters below for more details.\n",
"# https://www.tensorflow.org/lite/models/convert/metadata#normalization_and_quantization_parameters)\n",
"_INPUT_NORM_MEAN = 127.5\n",
"_INPUT_NORM_STD = 127.5\n",
"\n",
"# Create the metadata writer.\n",
"writer = ImageSegmenterWriter.create_for_inference(\n",
" writer_utils.load_file(_MODEL_PATH), [_INPUT_NORM_MEAN], [_INPUT_NORM_STD],\n",
" [_LABEL_FILE])\n",
"\n",
"# Verify the metadata generated by metadata writer.\n",
"print(writer.get_metadata_json())\n",
"\n",
"# Populate the metadata into the model.\n",
"writer_utils.save_file(writer.populate(), _SAVE_TO_PATH)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "NnvM80e7AG-h"
},
"source": [
"<a name=nl_classifiers></a>\n",
"###Natural language classifiers"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dfOPhFwOAG-k"
},
"source": [
"See the [natural language classifier model compatibility requirements](https://www.tensorflow.org/lite/inference_with_metadata/task_library/nl_classifier#model_compatibility_requirements) for more details about the supported model format."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "WMJ7tvuwAG-k"
},
"source": [
"Step 1: Import the required packages."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "_FGVyb2iAG-k"
},
"outputs": [],
"source": [
"from tflite_support.metadata_writers import nl_classifier\n",
"from tflite_support.metadata_writers import metadata_info\n",
"from tflite_support.metadata_writers import writer_utils"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "iIg7rATpAG-l"
},
"source": [
"Step 2: Download the example natural language classifier, [movie_review.tflite](https://github.com/tensorflow/tflite-support/blob/master/tensorflow_lite_support/metadata/python/tests/testdata/nl_classifier/movie_review.tflite), the [label file](https://github.com/tensorflow/tflite-support/blob/master/tensorflow_lite_support/metadata/python/tests/testdata/nl_classifier/labels.txt), and the [vocab file](https://storage.googleapis.com/download.tensorflow.org/models/tflite_support/nl_classifier/vocab.txt)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "TzuQcti2AG-l"
},
"outputs": [],
"source": [
"!curl -L https://github.com/tensorflow/tflite-support/raw/master/tensorflow_lite_support/metadata/python/tests/testdata/nl_classifier/movie_review.tflite -o movie_review.tflite\n",
"!curl -L https://github.com/tensorflow/tflite-support/raw/master/tensorflow_lite_support/metadata/python/tests/testdata/nl_classifier/labels.txt -o movie_review_labels.txt\n",
"!curl -L https://storage.googleapis.com/download.tensorflow.org/models/tflite_support/nl_classifier/vocab.txt -o movie_review_vocab.txt"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "BWxUtHdeAG-m"
},
"source": [
"Step 3: Create metadata writer and populate."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "NGPWzRuHAG-m"
},
"outputs": [],
"source": [
"NLClassifierWriter = nl_classifier.MetadataWriter\n",
"_MODEL_PATH = \"movie_review.tflite\"\n",
"# Task Library expects label files and vocab files that are in the same formats\n",
"# as the ones below.\n",
"_LABEL_FILE = \"movie_review_labels.txt\"\n",
"_VOCAB_FILE = \"movie_review_vocab.txt\"\n",
"# NLClassifier supports tokenize input string using the regex tokenizer. See\n",
"# more details about how to set up RegexTokenizer below:\n",
"# https://github.com/tensorflow/tflite-support/blob/master/tensorflow_lite_support/metadata/python/metadata_writers/metadata_info.py#L130\n",
"_DELIM_REGEX_PATTERN = r\"[^\\w\\']+\"\n",
"_SAVE_TO_PATH = \"moview_review_metadata.tflite\"\n",
"\n",
"# Create the metadata writer.\n",
"writer = nl_classifier.MetadataWriter.create_for_inference(\n",
" writer_utils.load_file(_MODEL_PATH),\n",
" metadata_info.RegexTokenizerMd(_DELIM_REGEX_PATTERN, _VOCAB_FILE),\n",
" [_LABEL_FILE])\n",
"\n",
"# Verify the metadata generated by metadata writer.\n",
"print(writer.get_metadata_json())\n",
"\n",
"# Populate the metadata into the model.\n",
"writer_utils.save_file(writer.populate(), _SAVE_TO_PATH)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "qv0WDnzW711f"
},
"source": [
"<a name=audio_classifiers></a>\n",
"### Audio classifiers"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "xqP7X8jww8pL"
},
"source": [
"See the [audio classifier model compatibility requirements](https://www.tensorflow.org/lite/inference_with_metadata/task_library/audio_classifier#model_compatibility_requirements) for more details about the supported model format."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7RToKepxw8pL"
},
"source": [
"Step 1: Import the required packages."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "JjddvTXKw8pL"
},
"outputs": [],
"source": [
"from tflite_support.metadata_writers import audio_classifier\n",
"from tflite_support.metadata_writers import metadata_info\n",
"from tflite_support.metadata_writers import writer_utils"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ar418rH6w8pL"
},
"source": [
"Step 2: Download the example audio classifier, [yamnet.tflite](https://github.com/tensorflow/tflite-support/blob/master/tensorflow_lite_support/metadata/python/tests/testdata/audio_classifier/yamnet_wavin_quantized_mel_relu6.tflite), and the [label file](https://github.com/tensorflow/tflite-support/blob/master/tensorflow_lite_support/metadata/python/tests/testdata/audio_classifier/yamnet_521_labels.txt)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "5eQY6znmw8pM"
},
"outputs": [],
"source": [
"!curl -L https://github.com/tensorflow/tflite-support/raw/master/tensorflow_lite_support/metadata/python/tests/testdata/audio_classifier/yamnet_wavin_quantized_mel_relu6.tflite -o yamnet.tflite\n",
"!curl -L https://github.com/tensorflow/tflite-support/raw/master/tensorflow_lite_support/metadata/python/tests/testdata/audio_classifier/yamnet_521_labels.txt -o yamnet_labels.txt\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "1TYP5w0Ew8pM"
},
"source": [
"Step 3: Create metadata writer and populate."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "MDlSczBQw8pM"
},
"outputs": [],
"source": [
"AudioClassifierWriter = audio_classifier.MetadataWriter\n",
"_MODEL_PATH = \"yamnet.tflite\"\n",
"# Task Library expects label files that are in the same format as the one below.\n",
"_LABEL_FILE = \"yamnet_labels.txt\"\n",
"# Expected sampling rate of the input audio buffer.\n",
"_SAMPLE_RATE = 16000\n",
"# Expected number of channels of the input audio buffer. Note that Task library only\n",
"# supports single channel so far.\n",
"_CHANNELS = 1\n",
"_SAVE_TO_PATH = \"yamnet_metadata.tflite\"\n",
"\n",
"# Create the metadata writer.\n",
"writer = AudioClassifierWriter.create_for_inference(\n",
" writer_utils.load_file(_MODEL_PATH), _SAMPLE_RATE, _CHANNELS, [_LABEL_FILE])\n",
"\n",
"# Verify the metadata generated by the metadata writer.\n",
"print(writer.get_metadata_json())\n",
"\n",
"# Populate the metadata into the model.\n",
"writer_utils.save_file(writer.populate(), _SAVE_TO_PATH)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "YoRLs84yNAJR"
},
"source": [
"## Create Model Metadata with semantic information"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "cxXsOBknOGJ2"
},
"source": [
"You can fill in more descriptive information about the model and each tensor through the Metadata Writer API to help improve model understanding. It can be done through the 'create_from_metadata_info' method in each metadata writer. In general, you can fill in data through the parameters of 'create_from_metadata_info', i.e. `general_md`, `input_md`, and `output_md`. See the example below to create a rich Model Metadata for image classifiers."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Q-LW6nrcQ9lv"
},
"source": [
"Step 1: Import the required packages."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "KsL_egYcRGw3"
},
"outputs": [],
"source": [
"from tflite_support.metadata_writers import image_classifier\n",
"from tflite_support.metadata_writers import metadata_info\n",
"from tflite_support.metadata_writers import writer_utils\n",
"from tflite_support import metadata_schema_py_generated as _metadata_fb"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0UWck_8uRboF"
},
"source": [
"Step 2: Download the example image classifier, [mobilenet_v2_1.0_224.tflite](https://github.com/tensorflow/tflite-support/raw/master/tensorflow_lite_support/metadata/python/tests/testdata/image_classifier/mobilenet_v2_1.0_224.tflite), and the [label file](https://github.com/tensorflow/tflite-support/raw/master/tensorflow_lite_support/metadata/python/tests/testdata/image_classifier/labels.txt)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "TqJ-jh-PRVdk"
},
"outputs": [],
"source": [
"!curl -L https://github.com/tensorflow/tflite-support/raw/master/tensorflow_lite_support/metadata/python/tests/testdata/image_classifier/mobilenet_v2_1.0_224.tflite -o mobilenet_v2_1.0_224.tflite\n",
"!curl -L https://github.com/tensorflow/tflite-support/raw/master/tensorflow_lite_support/metadata/python/tests/testdata/image_classifier/labels.txt -o mobilenet_labels.txt"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "r4I5wJMQRxzb"
},
"source": [
"Step 3: Create model and tensor information."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "urd7HDuaR_HC"
},
"outputs": [],
"source": [
"model_buffer = writer_utils.load_file(\"mobilenet_v2_1.0_224.tflite\")\n",
"\n",
"# Create general model information.\n",
"general_md = metadata_info.GeneralMd(\n",
" name=\"ImageClassifier\",\n",
" version=\"v1\",\n",
" description=(\"Identify the most prominent object in the image from a \"\n",
" \"known set of categories.\"),\n",
" author=\"TensorFlow Lite\",\n",
" licenses=\"Apache License. Version 2.0\")\n",
"\n",
"# Create input tensor information.\n",
"input_md = metadata_info.InputImageTensorMd(\n",
" name=\"input image\",\n",
" description=(\"Input image to be classified. The expected image is \"\n",
" \"128 x 128, with three channels (red, blue, and green) per \"\n",
" \"pixel. Each element in the tensor is a value between min and \"\n",
" \"max, where (per-channel) min is [0] and max is [255].\"),\n",
" norm_mean=[127.5],\n",
" norm_std=[127.5],\n",
" color_space_type=_metadata_fb.ColorSpaceType.RGB,\n",
" tensor_type=writer_utils.get_input_tensor_types(model_buffer)[0])\n",
"\n",
"# Create output tensor information.\n",
"output_md = metadata_info.ClassificationTensorMd(\n",
" name=\"probability\",\n",
" description=\"Probabilities of the 1001 labels respectively.\",\n",
" label_files=[\n",
" metadata_info.LabelFileMd(file_path=\"mobilenet_labels.txt\",\n",
" locale=\"en\")\n",
" ],\n",
" tensor_type=writer_utils.get_output_tensor_types(model_buffer)[0])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "N5aL5Uxkf4aO"
},
"source": [
"Step 4: Create metadata writer and populate."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "_iWIwdqEf_mr"
},
"outputs": [],
"source": [
"ImageClassifierWriter = image_classifier.MetadataWriter\n",
"# Create the metadata writer.\n",
"writer = ImageClassifierWriter.create_from_metadata_info(\n",
" model_buffer, general_md, input_md, output_md)\n",
"\n",
"# Verify the metadata generated by the metadata writer.\n",
"print(writer.get_metadata_json())\n",
"\n",
"# Populate the metadata into the model.\n",
"writer_utils.save_file(writer.populate(), _SAVE_TO_PATH)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "z78vuu6np5sb"
},
"source": [
"## Read the metadata populated to your model."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DnWt-4oOselo"
},
"source": [
"You can display the metadata and associated files in a TFLite model through the following code:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "5D13YPUsp5VT"
},
"outputs": [],
"source": [
"from tflite_support import metadata\n",
"\n",
"displayer = metadata.MetadataDisplayer.with_model_file(\"mobilenet_v2_1.0_224_metadata.tflite\")\n",
"print(\"Metadata populated:\")\n",
"print(displayer.get_metadata_json())\n",
"\n",
"print(\"Associated file(s) populated:\")\n",
"for file_name in displayer.get_packed_associated_file_list():\n",
" print(\"file name: \", file_name)\n",
" print(\"file content:\")\n",
" print(displayer.get_associated_file_buffer(file_name))"
]
}
],
"metadata": {
"colab": {
"collapsed_sections": [
"Mq-riZs-TJGt"
],
"name": "Metadata Writer tutorial",
"private_outputs": true,
"provenance": [],
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
},
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
tensorflow/lite/g3doc/models/convert/operation_fusion.md
+348
View File
@@ -0,0 +1,348 @@
# TensorFlow operation fusion
## Overview
This page describes the design and steps needed to convert composite operations
in TensorFlow to fused operations in TensorFlow Lite. This infrastructure is
general purpose and supports conversion of any composite operation in TensorFlow
to a corresponding fused operation in TensorFlow Lite.
An example use of this infrastructure is TensorFlow RNN operation fusion to
TensorFlow Lite, as detailed
[here](https://www.tensorflow.org/lite/models/convert/rnn).
### What are fused operations
![drawing](../../images/convert/op_fusion_banner.jpg)
TensorFlow operations can either be primitive ops e.g.
[tf.add](https://www.tensorflow.org/api_docs/python/tf/math/add) or they can be
composed from other primitive operations e.g.
[tf.einsum](https://www.tensorflow.org/api_docs/python/tf/einsum). A primitive
operation shows up as a single node in the TensorFlow graph while.a composite
operation is a collection of nodes in the TensorFlow graph. Executing a
composite operation is equivalent to executing each of its constituent primitive
operations.
A fused operation corresponds to a single operation that subsumes all the
computation performed by each primitive operation within the corresponding
composite operation.
### Benefits of fused operations
Fused operations exist to maximize the performance of their underlying kernel
implementations, by optimizing the overall computation and reducing memory
footprint. This is very valuable, especially for low-latency inference workloads
and resource constrained mobile platforms.
Fused operations also provide a higher level interface to define complex
transformations like quantization, which would otherwise be infeasible or very
hard to do at a more granular level.
TensorFlow Lite has many instances of fused operations for the reasons
articulated above. These fused operations typically correspond to composite
operations in the source TensorFlow program. Examples of composite operations in
TensorFlow that are implemented as a single fused operation in TensorFlow Lite
include various RNN operations like Unidirectional and Bidirectional sequence
LSTM, convolution (conv2d, bias add, relu), fully connected (matmul, bias add,
relu) and more. In TensorFlow Lite, LSTM quantization is currently only
implemented in the fused LSTM operations.
### Challenges with fused operations
Converting composite operations from TensorFlow to fused operations in
TensorFlow Lite is a hard problem. This is because:
1. Composite operations are represented in the TensorFlow graph as a set of
primitive operations without a well defined boundary. It can be very
challenging to identify (e.g. via pattern matching) the sub-graph
corresponding to such a composite operation.
1. There may be more than one TensorFlow implementation targeting a fused
TensorFlow Lite operation. For example, there are many LSTM implementations
in TensorFlow (Keras, Babelfish/lingvo etc) and each of these is composed of
different primitive operations but they all could still be converted to the
same fused LSTM operation in TensorFlow Lite.
As such, conversion of fused operations has proven quite challenging.
## Converting from composite op to a TFLite custom operation (recommended)
### Wrap the composite operation in a `tf.function`
In many cases, some part of the model can be mapped to a single operation in
TFLite. This can help with performance when writing an optimized implementation
for specific operations. To be able to create a fused operation in TFLite,
identify the part of the graph that represents a fused operation and wrap it in
a `tf.function` with "experimental_implements" attribute to a `tf.function`,
which has attribute value `tfl_fusable_op` with value `true`. If the custom
operation takes attributes then pass them as part of the same
"experimental_implements".
Example,
```python
def get_implements_signature():
implements_signature = [
# 'name' will be used as a name for the operation.
'name: "my_custom_fused_op"',
# attr "tfl_fusable_op" is required to be set with true value.
'attr {key: "tfl_fusable_op" value { b: true } }',
# Example attribute "example_option" that the op accepts.
'attr {key: "example_option" value { i: %d } }' % 10
]
return ' '.join(implements_signature)
@tf.function(experimental_implements=get_implements_signature())
def my_custom_fused_op(input_1, input_2):
# An empty function that represents pre/post processing example that
# is not represented as part of the Tensorflow graph.
output_1 = tf.constant(0.0, dtype=tf.float32, name='first_output')
output_2 = tf.constant(0.0, dtype=tf.float32, name='second_output')
return output_1, output_2
class TestModel(tf.Module):
def __init__(self):
super(TestModel, self).__init__()
self.conv_1 = tf.keras.layers.Conv2D(filters=1, kernel_size=(3, 3))
self.conv_2 = tf.keras.layers.Conv2D(filters=1, kernel_size=(3, 3))
@tf.function(input_signature=[
tf.TensorSpec(shape=[1, 28, 28, 3], dtype=tf.float32),
tf.TensorSpec(shape=[1, 28, 28, 3], dtype=tf.float32),
])
def simple_eval(self, input_a, input_b):
return my_custom_fused_op(self.conv_1(input_a), self.conv_2(input_b))
```
Note that you don't need to set `allow_custom_ops` on the converter as
`tfl_fusable_op` attribute imply this already.
### Implement custom op and register with TFLite Interpreter
Implement your fused operation as a TFLite Custom operation - see
[instructions](https://www.tensorflow.org/lite/guide/ops_custom).
Note that, the name to register the op with should be similar to the name
specified in the `name` attribute in the implements signature.
An example for the op in the example is
```c++
TfLiteRegistration reg = {};
// This name must match the name specified in the implements signature.
static constexpr char kOpName[] = "my_custom_fused_op";
reg.custom_name = kOpName;
reg.prepare = [](TfLiteContext* context, TfLiteNode* node) -> TfLiteStatus {
// Add your code.
return kTfLiteOk;
};
reg.invoke = [](TfLiteContext* context, TfLiteNode* node) -> TfLiteStatus {
// Add your code.
return kTfLiteOk;
};
reg.builtin_code = kTfLiteCustom;
resolver->AddCustom(kOpName, &reg);
```
## Converting from composite to fused operation (Advanced)
The overall architecture for converting TensorFlow composite operations to
TensorFlow Lite fused operations is below:
![drawing](../../images/convert/op_fusion.png)
### Wrap the composite operation in a `tf.function`
In the TensorFlow model source code, identify and abstract out the composite
operation into a `tf.function` with the
[experimental\_implements](https://github.com/tensorflow/tensorflow/blob/c11d5d8881fd927165eeb09fd524a80ebaf009f2/tensorflow/python/eager/def_function.py#L470)
function annotation. See an example of [embedding lookup](#composing_ops). The
function defines the interface and its arguments should be used to implement the
conversion logic.
### Write conversion code
The conversion code is written per the interface of the function with the
`implements` annotation. See an example fusion for
[embedding lookup](#fusion_code). Conceptually, the conversion code replaces the
composite implementation of this interface with the fused one.
In the prepare-composite-functions pass, plugin in your
[conversion code](https://github.com/tensorflow/tensorflow/blob/c11d5d8881fd927165eeb09fd524a80ebaf009f2/tensorflow/compiler/mlir/lite/transforms/prepare_composite_functions_tf.cc#L115).
In more advanced usages, it is possible to implement complex transformations of
the composite operation's operands in order to derive the operands of the fused
operation. See
[Keras LSTM](https://github.com/tensorflow/tensorflow/blob/1099faa8d6a941ef44d09ed8c372ff0ffda94112/tensorflow/compiler/mlir/lite/utils/lstm_utils.cc#L627).
conversion code as an example.
### Convert to TensorFlow Lite
Use the
[TFLiteConverter.from_saved_model](https://www.tensorflow.org/api_docs/python/tf/lite/TFLiteConverter#from_saved_model)
API to convert to TensorFlow Lite.
## Under the hood
<a id="under_the_hood"></a>
We now describe high level details of the overall design in converting to fused
operations in TensorFlow Lite.
### Composing operations in TensorFlow
<a id="composing_ops"></a>
The use of `tf.function` with the
[experimental\_implements](https://github.com/tensorflow/tensorflow/blob/c11d5d8881fd927165eeb09fd524a80ebaf009f2/tensorflow/python/eager/def_function.py#L470)
function attribute allows users to explicitly compose new operations using
TensorFlow primitive operations and specify the interface that the resultant
composite operation implements. This is very useful as it provides:
1. A well-defined boundary for the composite operation in the underlying
TensorFlow graph.
1. Explicitly specify the interface that this operation implements. The
arguments of the `tf.function` correspond to the arguments of this
interface.
As an example, let’s consider a composite operation defined to implement
embedding lookup. This maps to a fused operation in TensorFlow Lite.
```python
@tf.function(
experimental_implements="embedding_lookup")
def EmbFprop(embs, ids_vec):
"""Embedding forward prop.
Effectively, it computes:
num = size of ids_vec
rets = zeros([num, embedding dim])
for i in range(num):
rets[i, :] = embs[ids_vec[i], :]
return rets
Args:
embs: The embedding matrix.
ids_vec: A vector of int32 embedding ids.
Returns:
The result of embedding lookups. A matrix of shape
[num ids in ids_vec, embedding dims].
"""
num = tf.shape(ids_vec)[0]
rets = inplace_ops.empty([num] + emb_shape_suf, py_utils.FPropDtype(p))
def EmbFpropLoop(i, embs, ids_vec, rets):
# row_id = ids_vec[i]
row_id = tf.gather(ids_vec, i)
# row = embs[row_id]
row = tf.reshape(tf.gather(embs, row_id), [1] + emb_shape_suf)
# rets[i] = row
rets = inplace_ops.alias_inplace_update(rets, [i], row)
return embs, ids_vec, rets
_, _, rets = functional_ops.For(
start=0,
limit=num,
delta=1,
inputs=[embs, ids_vec, rets],
body=EmbFpropLoop,
rewrite_with_while=compiled)
if len(weight_shape) > 2:
rets = tf.reshape(rets, [num, symbolic.ToStatic(p.embedding_dim)])
return rets
```
By making models use composite operations via `tf.function` as illustrated
above, it becomes possible to build a general infrastructure to **identify and
convert** such operations to fused TensorFlow Lite operations.
### Extending the TensorFlow Lite converter
The TensorFlow Lite converter that was released earlier this year only supported
importing TensorFlow models as a graph with all variables replaced with their
corresponding constant values. This does not work for operation fusion since
such graphs have all functions inlined so that the variables can be turned into
constants.
In order to leverage the `tf.function` with the `experimental_implements`
feature during the conversion process, the functions need to be preserved until
later in the conversion process.
As such, we implemented a new workflow of importing and converting TensorFlow
models in the converter to support the composite operation fusion use case.
Specifically, the new features added are:
1. Importing TensorFlow
[saved models into MLIR](https://github.com/tensorflow/tensorflow/blob/1099faa8d6a941ef44d09ed8c372ff0ffda94112/tensorflow/compiler/mlir/tensorflow/translate/import_model.cc#L3748)
1. [fuse composite operations](https://github.com/tensorflow/tensorflow/blob/1099faa8d6a941ef44d09ed8c372ff0ffda94112/tensorflow/compiler/mlir/lite/transforms/prepare_composite_functions_tf.cc#L103)
1. [variable mutability analysis](https://github.com/tensorflow/tensorflow/blob/1099faa8d6a941ef44d09ed8c372ff0ffda94112/tensorflow/compiler/mlir/tensorflow/transforms/optimize_global_tensors.cc#L43)
1. [freeze all read-only variables](https://github.com/tensorflow/tensorflow/blob/1099faa8d6a941ef44d09ed8c372ff0ffda94112/tensorflow/compiler/mlir/tensorflow/transforms/freeze_global_tensors.cc#L44)
This allows us to perform operation fusion using the functions representing the
composite operations prior to function inlining and variable freezing.
### Implementing operation fusion
Let’s look at the operation fusion pass in more detail. This pass does the
following:
1. Loop through all functions in the MLIR module.
1. If a function has the tf.\_implements attribute, based on the attribute
value, calls the appropriate operation fusion utility.
1. The operation fusion utility operates on the function’s operands and
attributes (which serve as the interface for the conversion) and replaces
the body of the function with an equivalent function body containing the
fused operation.
1. In many cases, the replaced body will contain operations other than the
fused operation. These correspond to some static transforms on the
function’s operands in order to obtain the operands of the fused operation.
Since these computations can all be constant folded away, they would not be
present in the exported flatbuffer where only the fused operation would
exist.
Here is code snippet from the pass showing the main workflow:
```
void PrepareCompositeFunctionsPass::ConvertTFImplements(FuncOp func,
StringAttr attr) {
if (attr.getValue() == "embedding_lookup") {
func.eraseBody();
func.addEntryBlock();
// Convert the composite embedding_lookup function body to a
// TFLite fused embedding_lookup op.
ConvertEmbeddedLookupFunc convert_embedded_lookup(func);
if (failed(convert_embedded_lookup.VerifySignature())) {
return signalPassFailure();
}
convert_embedded_lookup.RewriteFunc();
} else if (attr.getValue() == mlir::TFL::kKerasLstm) {
func.eraseBody();
func.addEntryBlock();
OpBuilder builder(func.getBody());
if (failed(ConvertKerasLSTMLayer(func, &builder))) {
return signalPassFailure();
}
} else if (.....) /* Other fusions can plug in here */
}
```
Here is code snippet showing mapping this composite operation to a fused
operation in TensorFlow Lite leveraging the function as a conversion interface.
<a id="fusion_code"></a>
```c++
void RewriteFunc() {
Value lookup = func_.getArgument(1);
Value value = func_.getArgument(0);
auto output_type = func_.getType().getResult(0);
OpBuilder builder(func_.getBody());
auto op = builder.create<mlir::TFL::EmbeddingLookupOp>(
func_.getLoc(), output_type, lookup, value);
builder.create<mlir::ReturnOp>(func_.getLoc(), op.getResult());
}
```
tensorflow/lite/g3doc/models/convert/rnn.md
+198
View File
@@ -0,0 +1,198 @@
# TensorFlow RNN conversion to TensorFlow Lite
## Overview
TensorFlow Lite supports converting TensorFlow RNN models to TensorFlow Lite’s
fused LSTM operations. Fused operations exist to maximize the performance of
their underlying kernel implementations, as well as provide a higher level
interface to define complex transformations like quantizatization.
Since there are many variants of RNN APIs in TensorFlow, our approach has been
two fold:
1. Provide **native support for standard TensorFlow RNN APIs** like Keras LSTM.
This is the recommended option.
1. Provide an **interface** **into the conversion infrastructure for**
**user-defined** **RNN implementations** to plug in and get converted to
TensorFlow Lite. We provide a couple of out of box examples of such
conversion using lingvo’s
[LSTMCellSimple](https://github.com/tensorflow/tensorflow/blob/82abf0dbf316526cd718ae8cd7b11cfcb805805e/tensorflow/compiler/mlir/lite/transforms/prepare_composite_functions_tf.cc#L130)
and
[LayerNormalizedLSTMCellSimple](https://github.com/tensorflow/tensorflow/blob/c11d5d8881fd927165eeb09fd524a80ebaf009f2/tensorflow/compiler/mlir/lite/transforms/prepare_composite_functions_tf.cc#L137)
RNN interfaces.
## Converter API
The feature is part of TensorFlow 2.3 release. It is also available through the
[tf-nightly](https://pypi.org/project/tf-nightly/) pip or from head.
This conversion functionality is available when converting to TensorFlow Lite
via a SavedModel or from the Keras model directly. See example usages.
### From saved model
<a id="from_saved_model"></a>
```
# build a saved model. Here concrete_function is the exported function
# corresponding to the TensorFlow model containing one or more
# Keras LSTM layers.
saved_model, saved_model_dir = build_saved_model_lstm(...)
saved_model.save(saved_model_dir, save_format="tf", signatures=concrete_func)
# Convert the model.
converter = TFLiteConverter.from_saved_model(saved_model_dir)
tflite_model = converter.convert()
```
### From Keras model
```
# build a Keras model
keras_model = build_keras_lstm(...)
# Convert the model.
converter = TFLiteConverter.from_keras_model(keras_model)
tflite_model = converter.convert()
```
## Example
Keras LSTM to TensorFlow Lite
[Colab](https://colab.research.google.com/github/tensorflow/tensorflow/blob/master/tensorflow/lite/examples/experimental_new_converter/Keras_LSTM_fusion_Codelab.ipynb)
illustrates the end to end usage with the TensorFlow Lite interpreter.
## TensorFlow RNNs APIs supported
<a id="rnn_apis"></a>
### Keras LSTM conversion (recommended)
We support out-of-the-box conversion of Keras LSTM to TensorFlow Lite. For
details on how this works please refer to the
[Keras LSTM interface](https://github.com/tensorflow/tensorflow/blob/35a3ab91b42503776f428bda574b74b9a99cd110/tensorflow/python/keras/layers/recurrent_v2.py#L1238)<span style="text-decoration:space;">
</span>and to the conversion logic
[here](https://github.com/tensorflow/tensorflow/blob/35a3ab91b42503776f428bda574b74b9a99cd110/tensorflow/compiler/mlir/lite/utils/lstm_utils.cc#L627).
Also important is to highlight the TensorFlow Lite’s LSTM contract with respect
to the Keras operation definition:
1. The dimension 0 of the **input** tensor is the batch size.
1. The dimension 0 of the **recurrent\_weight** tensor is the number of
outputs.
1. The **weight** and **recurrent\_kernel** tensors are transposed.
1. The transposed weight, transposed recurrent\_kernel and **bias** tensors are
split into 4 equal sized tensors along the dimension 0. These correspond to
**input gate, forget gate, cell, and output gate**.
#### Keras LSTM Variants
##### Time major
Users may choose time-major or no time-major. Keras LSTM adds a time-major
attribute in the function def attributes. For Unidirectional sequence LSTM, we
can simply map to unidirecional\_sequence\_lstm's
[time major attribute](https://github.com/tensorflow/tensorflow/blob/35a3ab91b42503776f428bda574b74b9a99cd110/tensorflow/compiler/mlir/lite/ir/tfl_ops.td#L3902).
##### BiDirectional LSTM
Bidirectional LSTM can be implemented with two Keras LSTM layers, one for
forward and one for backward, see examples
[here](https://github.com/tensorflow/tensorflow/blob/35a3ab91b42503776f428bda574b74b9a99cd110/tensorflow/python/keras/layers/wrappers.py#L382).
Once we see the go\_backward attribute, we recognize it as backward LSTM, then
we group forward & backward LSTM together. **This is future work.** Currently,
this creates two UnidirectionalSequenceLSTM operations in the TensorFlow Lite
model.
### User-defined LSTM conversion examples
TensorFlow Lite also provides a way to convert user defined LSTM
implementations. Here we use Lingvo’s LSTM as an example of how that can be
implemented. For details please refer to the
[lingvo.LSTMCellSimple interface](https://github.com/tensorflow/lingvo/blob/91a4609dbc2579748a95110eda59c66d17c594c5/lingvo/core/rnn_cell.py#L228)
and the conversion logic
[here](https://github.com/tensorflow/tensorflow/blob/82abf0dbf316526cd718ae8cd7b11cfcb805805e/tensorflow/compiler/mlir/lite/transforms/prepare_composite_functions_tf.cc#L130).
We also provide an example for another of Lingvo’s LSTM definitions in
[lingvo.LayerNormalizedLSTMCellSimple interface](https://github.com/tensorflow/lingvo/blob/91a4609dbc2579748a95110eda59c66d17c594c5/lingvo/core/rnn_cell.py#L1173)
and its conversion logic
[here](https://github.com/tensorflow/tensorflow/blob/c11d5d8881fd927165eeb09fd524a80ebaf009f2/tensorflow/compiler/mlir/lite/transforms/prepare_composite_functions_tf.cc#L137).
## “Bring your own TensorFlow RNN” to TensorFlow Lite
If a user's RNN interface is different from the standard supported ones, there
are a couple of options:
**Option 1:** Write adapter code in TensorFlow python to adapt the RNN interface
to the Keras RNN interface. This means a tf.function with
[tf\_implements annotation](https://github.com/tensorflow/community/pull/113) on
the generated RNN interface’s function that is identical to the one generated by
the Keras LSTM layer. After this, the same conversion API used for Keras LSTM
will work.
**Option 2:** If the above is not possible (e.g. the Keras LSTM is missing some
functionality that is currently exposed by TensorFlow Lite’s fused LSTM op like
layer normalization), then extend the TensorFlow Lite converter by writing
custom conversion code and plug it into the prepare-composite-functions
MLIR-pass
[here](https://github.com/tensorflow/tensorflow/blob/c11d5d8881fd927165eeb09fd524a80ebaf009f2/tensorflow/compiler/mlir/lite/transforms/prepare_composite_functions_tf.cc#L115).
The function’s interface should be treated like an API contract and should
contain the arguments needed to convert to fused TensorFlow Lite LSTM
operations - i.e. input, bias, weights, projection, layer normalization, etc. It
is preferable for the tensors passed as arguments to this function to have known
rank (i.e. RankedTensorType in MLIR). This makes it much easier to write
conversion code that can assume these tensors as RankedTensorType and helps
transform them to ranked tensors corresponding to the fused TensorFlow Lite
operator’s operands.
A complete example of such conversion flow is Lingvo’s LSTMCellSimple to
TensorFlow Lite conversion.
The LSTMCellSimple in Lingvo is defined
[here](https://github.com/tensorflow/lingvo/blob/91a4609dbc2579748a95110eda59c66d17c594c5/lingvo/core/rnn_cell.py#L228).
Models trained with this LSTM cell can be converted to TensorFlow Lite as
follows:
1. Wrap all uses of LSTMCellSimple in a tf.function with a tf\_implements
annotation that is labelled as such (e.g. lingvo.LSTMCellSimple would be a
good annotation name here). Make sure the tf.function that is generated
matches the interface of the function expected in the conversion code. This
is a contract between the model author adding the annotation and the
conversion code.
1. Extend the prepare-composite-functions pass to plug in a custom composite op
to TensorFlow Lite fused LSTM op conversion. See
[LSTMCellSimple](https://github.com/tensorflow/tensorflow/blob/82abf0dbf316526cd718ae8cd7b11cfcb805805e/tensorflow/compiler/mlir/lite/transforms/prepare_composite_functions_tf.cc#L130)
conversion code.
The conversion contract:
1. **Weight** and **projection** tensors are transposed.
1. The **{input, recurrent}** to **{cell, input gate, forget gate, output
gate}** are extracted by slicing the transposed weight tensor.
1. The **{bias}** to **{cell, input gate, forget gate, output gate}** are
extracted by slicing the bias tensor.
1. The **projection** is extracted by slicing the transposed projection tensor.
1. Similar conversion is written for
[LayerNormalizedLSTMCellSimple](https://github.com/tensorflow/tensorflow/blob/c11d5d8881fd927165eeb09fd524a80ebaf009f2/tensorflow/compiler/mlir/lite/transforms/prepare_composite_functions_tf.cc#L137).
1. The rest of the TensorFlow Lite conversion infrastructure, including all the
[MLIR passes](https://github.com/tensorflow/tensorflow/blob/35a3ab91b42503776f428bda574b74b9a99cd110/tensorflow/compiler/mlir/lite/tf_tfl_passes.cc#L57)
defined as well as the final export to TensorFlow Lite flatbuffer can be
reused.
## Known issues/limitations
1. Currently there is support only for converting stateless Keras LSTM (default
behavior in Keras). Stateful Keras LSTM conversion is future work.
1. It is still possible to model a stateful Keras LSTM layer using the
underlying stateless Keras LSTM layer and managing the state explicitly in
the user program. Such a TensorFlow program can still be converted to
TensorFlow Lite using the feature being described here.
1. Bidirectional LSTM is currently modelled as two UnidirectionalSequenceLSTM
operations in TensorFlow Lite. This will be replaced with a single
BidirectionalSequenceLSTM op.