wehub/tensorflow--tensorflow
1
0
Fork 0
Code Issues Pull Requests Actions 5 Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution
cffconvert / validate (push) Has been skipped
Details
License Check / license-check (push) Failing after 2s
Details

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 12:14:16 +08:00
commit 8a852e4b4e
36502 changed files with 9277225 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tensorflow/lite/g3doc/performance/best_practices.md
+144
View File
@@ -0,0 +1,144 @@
# Performance best practices
Mobile and embedded devices have limited computational resources, so it is
important to keep your application resource efficient. We have compiled a list
of best practices and strategies that you can use to improve your TensorFlow
Lite model performance.
## Choose the best model for the task
Depending on the task, you will need to make a tradeoff between model complexity
and size. If your task requires high accuracy, then you may need a large and
complex model. For tasks that require less precision, it is better to use a
smaller model because they not only use less disk space and memory, but they are
also generally faster and more energy efficient. For example, graphs below show
accuracy and latency tradeoffs for some common image classification models.
![Graph of model size vs accuracy](../images/performance/model_size_vs_accuracy.png "Model Size vs Accuracy")
![Graph of accuracy vs latency](../images/performance/accuracy_vs_latency.png "Accuracy vs Latency")
One example of models optimized for mobile devices are
[MobileNets](https://arxiv.org/abs/1704.04861), which are optimized for mobile
vision applications.
[TensorFlow Hub](https://tfhub.dev/s?deployment-format=lite) lists several other
models that have been optimized specifically for mobile and embedded devices.
You can retrain the listed models on your own dataset by using transfer
learning. Check out the transfer learning tutorials using TensorFlow Lite
[Model Maker](../models/modify/model_maker/).
## Profile your model
Once you have selected a candidate model that is right for your task, it is a
good practice to profile and benchmark your model. TensorFlow Lite
[benchmarking tool](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/tools/benchmark)
has a built-in profiler that shows per operator profiling statistics. This can
help in understanding performance bottlenecks and which operators dominate the
computation time.
You can also use
[TensorFlow Lite tracing](measurement.md#trace-tensorflow-lite-internals-in-android)
to profile the model in your Android application, using standard Android system
tracing, and to visualize the operator invocations by time with GUI based
profiling tools.
## Profile and optimize operators in the graph
If a particular operator appears frequently in the model and, based on
profiling, you find that the operator consumes the most amount of time, you can
look into optimizing that operator. This scenario should be rare as TensorFlow
Lite has optimized versions for most operators. However, you may be able to
write a faster version of a custom op if you know the constraints in which the
operator is executed. Check out the
[custom operators guide](../guide/ops_custom.md).
## Optimize your model
Model optimization aims to create smaller models that are generally faster and
more energy efficient, so that they can be deployed on mobile devices.
TensorFlow Lite supports multiple optimization techniques, such as quantization.
Check out the
[model optimization docs](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/performance/model_optimization.md)
for details.
## Tweak the number of threads
TensorFlow Lite supports multi-threaded kernels for many operators. You can
increase the number of threads and speed up execution of operators. Increasing
the number of threads will, however, make your model use more resources and
power.
For some applications, latency may be more important than energy efficiency. You
can increase the number of threads by setting the number of interpreter
[threads](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/interpreter.h#L346).
Multi-threaded execution, however, comes at the cost of increased performance
variability depending on what else is executed concurrently. This is
particularly the case for mobile apps. For example, isolated tests may show 2x
speed-up vs single-threaded, but, if another app is executing at the same time,
it may result in worse performance than single-threaded.
## Eliminate redundant copies
If your application is not carefully designed, there can be redundant copies
when feeding the input to and reading the output from the model. Make sure to
eliminate redundant copies. If you are using higher level APIs, like Java, make
sure to carefully check the documentation for performance caveats. For example,
the Java API is a lot faster if `ByteBuffers` are used as
[inputs](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/java/src/main/java/org/tensorflow/lite/Interpreter.java#L175).
## Profile your application with platform specific tools
Platform specific tools like
[Android profiler](https://developer.android.com/studio/profile/android-profiler)
and [Instruments](https://help.apple.com/instruments/mac/current/) provide a
wealth of profiling information that can be used to debug your app. Sometimes
the performance bug may be not in the model but in parts of application code
that interact with the model. Make sure to familiarize yourself with platform
specific profiling tools and best practices for your platform.
## Evaluate whether your model benefits from using hardware accelerators available on the device
TensorFlow Lite has added new ways to accelerate models with faster hardware
like GPUs, DSPs, and neural accelerators. Typically, these accelerators are
exposed through
[delegate](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/performance/delegates.md)
submodules that take over parts of the interpreter execution. TensorFlow Lite
can use delegates by:
* Using Android's
[Neural Networks API](https://developer.android.com/ndk/guides/neuralnetworks/).
You can utilize these hardware accelerator backends to improve the speed and
efficiency of your model. To enable the Neural Networks API, check out the
[NNAPI delegate](https://www.tensorflow.org/lite/android/delegates/nnapi)
guide.
* GPU delegate is available on Android and iOS, using OpenGL/OpenCL and Metal,
respectively. To try them out, see the
[GPU delegate tutorial](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/performance/gpu.md)
and
[documentation](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/performance/gpu.md#advanced-gpu-support).
* Hexagon delegate is available on Android. It leverages the Qualcomm Hexagon
DSP if it is available on the device. See the
[Hexagon delegate tutorial](https://www.tensorflow.org/lite/android/delegates/hexagon)
for more information.
* It is possible to create your own delegate if you have access to
non-standard hardware. See
[TensorFlow Lite delegates](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/performance/delegates.md)
for more information.
Be aware that some accelerators work better for different types of models. Some
delegates only support float models or models optimized in a specific way. It is
important to
[benchmark](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/performance/measurement.md)
each delegate to see if it is a good choice for your application. For example,
if you have a very small model, it may not be worth delegating the model to
either the NN API or the GPU. Conversely, accelerators are a great choice for
large models that have high arithmetic intensity.
## Need more help
The TensorFlow team is happy to help diagnose and address specific performance
issues you may be facing. Please file an issue on
[GitHub](https://github.com/tensorflow/tensorflow/issues) with details of the
issue.
tensorflow/lite/g3doc/performance/coreml_delegate.md
+305
View File
@@ -0,0 +1,305 @@
# Tensorflow Lite Core ML delegate
The TensorFlow Lite Core ML delegate enables running TensorFlow Lite models on
[Core ML framework](https://developer.apple.com/documentation/coreml), which
results in faster model inference on iOS devices.
Note: This delegate is in experimental (beta) phase. It is available from
TensorFlow Lite 2.4.0 and latest nightly releases.
Note: Core ML delegate supports Core ML version 2 and later.
**Supported iOS versions and devices:**
* iOS 12 and later. In the older iOS versions, Core ML delegate will
automatically fallback to CPU.
* By default, Core ML delegate will only be enabled on devices with A12 SoC
and later (iPhone Xs and later) to use Neural Engine for faster inference.
If you want to use Core ML delegate also on the older devices, please see
[best practices](#best-practices)
**Supported models**
The Core ML delegate currently supports float (FP32 and FP16) models.
## Trying the Core ML delegate on your own model
The Core ML delegate is already included in nightly release of TensorFlow lite
CocoaPods. To use Core ML delegate, change your TensorFlow lite pod to include
subspec `CoreML` in your `Podfile`.
Note: If you want to use C API instead of Objective-C API, you can include
`TensorFlowLiteC/CoreML` pod to do so.
```
target 'YourProjectName'
pod 'TensorFlowLiteSwift/CoreML', '~> 2.4.0' # Or TensorFlowLiteObjC/CoreML
```
OR
```
# Particularily useful when you also want to include 'Metal' subspec.
target 'YourProjectName'
pod 'TensorFlowLiteSwift', '~> 2.4.0', :subspecs => ['CoreML']
```
Note: Core ML delegate can also use C API for Objective-C code. Prior to
TensorFlow Lite 2.4.0 release, this was the only option.
<div>
<devsite-selector>
<section>
<h3>Swift</h3>
<p><pre class="prettyprint lang-swift">
let coreMLDelegate = CoreMLDelegate()
var interpreter: Interpreter
// Core ML delegate will only be created for devices with Neural Engine
if coreMLDelegate != nil {
interpreter = try Interpreter(modelPath: modelPath,
delegates: [coreMLDelegate!])
} else {
interpreter = try Interpreter(modelPath: modelPath)
}
</pre></p>
</section>
<section>
<h3>Objective-C</h3>
<p><pre class="prettyprint lang-objc">
// Import module when using CocoaPods with module support
@import TFLTensorFlowLite;
// Or import following headers manually
# import "tensorflow/lite/objc/apis/TFLCoreMLDelegate.h"
# import "tensorflow/lite/objc/apis/TFLTensorFlowLite.h"
// Initialize Core ML delegate
TFLCoreMLDelegate* coreMLDelegate = [[TFLCoreMLDelegate alloc] init];
// Initialize interpreter with model path and Core ML delegate
TFLInterpreterOptions* options = [[TFLInterpreterOptions alloc] init];
NSError* error = nil;
TFLInterpreter* interpreter = [[TFLInterpreter alloc]
initWithModelPath:modelPath
options:options
delegates:@[ coreMLDelegate ]
error:&amp;error];
if (error != nil) { /* Error handling... */ }
if (![interpreter allocateTensorsWithError:&amp;error]) { /* Error handling... */ }
if (error != nil) { /* Error handling... */ }
// Run inference ...
</pre></p>
</section>
<section>
<h3>C (Until 2.3.0)</h3>
<p><pre class="prettyprint lang-c">
#include "tensorflow/lite/delegates/coreml/coreml_delegate.h"
// Initialize interpreter with model
TfLiteModel* model = TfLiteModelCreateFromFile(model_path);
// Initialize interpreter with Core ML delegate
TfLiteInterpreterOptions* options = TfLiteInterpreterOptionsCreate();
TfLiteDelegate* delegate = TfLiteCoreMlDelegateCreate(NULL); // default config
TfLiteInterpreterOptionsAddDelegate(options, delegate);
TfLiteInterpreterOptionsDelete(options);
TfLiteInterpreter* interpreter = TfLiteInterpreterCreate(model, options);
TfLiteInterpreterAllocateTensors(interpreter);
// Run inference ...
/* ... */
// Dispose resources when it is no longer used.
// Add following code to the section where you dispose of the delegate
// (e.g. `dealloc` of class).
TfLiteInterpreterDelete(interpreter);
TfLiteCoreMlDelegateDelete(delegate);
TfLiteModelDelete(model);
</pre></p>
</section>
</devsite-selector>
</div>
## Best practices
### Using Core ML delegate on devices without Neural Engine
By default, Core ML delegate will only be created if the device has Neural
Engine, and will return `null` if the delegate is not created. If you want to
run Core ML delegate on other environments (for example, simulator), pass `.all`
as an option while creating delegate in Swift. On C++ (and Objective-C), you can
pass `TfLiteCoreMlDelegateAllDevices`. Following example shows how to do this:
<div>
<devsite-selector>
<section>
<h3>Swift</h3>
<p><pre class="prettyprint lang-swift">
var options = CoreMLDelegate.Options()
options.enabledDevices = .all
let coreMLDelegate = CoreMLDelegate(options: options)!
let interpreter = try Interpreter(modelPath: modelPath,
delegates: [coreMLDelegate])
</pre></p>
</section>
<section>
<h3>Objective-C</h3>
<p><pre class="prettyprint lang-objc">
TFLCoreMLDelegateOptions* coreMLOptions = [[TFLCoreMLDelegateOptions alloc] init];
coreMLOptions.enabledDevices = TFLCoreMLDelegateEnabledDevicesAll;
TFLCoreMLDelegate* coreMLDelegate = [[TFLCoreMLDelegate alloc]
initWithOptions:coreMLOptions];
// Initialize interpreter with delegate
</pre></p>
</section>
<section>
<h3>C</h3>
<p><pre class="prettyprint lang-c">
TfLiteCoreMlDelegateOptions options;
options.enabled_devices = TfLiteCoreMlDelegateAllDevices;
TfLiteDelegate* delegate = TfLiteCoreMlDelegateCreate(&amp;options);
// Initialize interpreter with delegate
</pre></p>
</section>
</devsite-selector>
</div>
### Using Metal(GPU) delegate as a fallback.
When the Core ML delegate is not created, alternatively you can still use
[Metal delegate](https://www.tensorflow.org/lite/performance/gpu#ios) to get
performance benefits. Following example shows how to do this:
<div>
<devsite-selector>
<section>
<h3>Swift</h3>
<p><pre class="prettyprint lang-swift">
var delegate = CoreMLDelegate()
if delegate == nil {
delegate = MetalDelegate() // Add Metal delegate options if necessary.
}
let interpreter = try Interpreter(modelPath: modelPath,
delegates: [delegate!])
</pre></p>
</section>
<section>
<h3>Objective-C</h3>
<p><pre class="prettyprint lang-objc">
TFLDelegate* delegate = [[TFLCoreMLDelegate alloc] init];
if (!delegate) {
// Add Metal delegate options if necessary
delegate = [[TFLMetalDelegate alloc] init];
}
// Initialize interpreter with delegate
</pre></p>
</section>
<section>
<h3>C</h3>
<p><pre class="prettyprint lang-c">
TfLiteCoreMlDelegateOptions options = {};
delegate = TfLiteCoreMlDelegateCreate(&amp;options);
if (delegate == NULL) {
// Add Metal delegate options if necessary
delegate = TFLGpuDelegateCreate(NULL);
}
// Initialize interpreter with delegate
</pre></p>
</section>
</devsite-selector>
</div>
The delegate creation logic reads device's machine id (e.g. iPhone11,1) to
determine its Neural Engine availability. See the
[code](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/delegates/coreml/coreml_delegate.mm)
for more detail. Alternatively, you can implement your own set of denylist
devices using other libraries such as
[DeviceKit](https://github.com/devicekit/DeviceKit).
### Using older Core ML version
Although iOS 13 supports Core ML 3, the model might work better when it is
converted with Core ML 2 model specification. The target conversion version is
set to the latest version by default, but you can change this by setting
`coreMLVersion` (in Swift, `coreml_version` in C API) in the delegate option to
older version.
## Supported ops
Following ops are supported by the Core ML delegate.
* Add
* Only certain shapes are broadcastable. In Core ML tensor layout,
following tensor shapes are broadcastable. `[B, C, H, W]`, `[B, C, 1,
1]`, `[B, 1, H, W]`, `[B, 1, 1, 1]`.
* AveragePool2D
* Concat
* Concatenation should be done along the channel axis.
* Conv2D
* Weights and bias should be constant.
* DepthwiseConv2D
* Weights and bias should be constant.
* FullyConnected (aka Dense or InnerProduct)
* Weights and bias (if present) should be constant.
* Only supports single-batch case. Input dimensions should be 1, except
the last dimension.
* Hardswish
* Logistic (aka Sigmoid)
* MaxPool2D
* MirrorPad
* Only 4D input with `REFLECT` mode is supported. Padding should be
constant, and is only allowed for H and W dimensions.
* Mul
* Only certain shapes are broadcastable. In Core ML tensor layout,
following tensor shapes are broadcastable. `[B, C, H, W]`, `[B, C, 1,
1]`, `[B, 1, H, W]`, `[B, 1, 1, 1]`.
* Pad and PadV2
* Only 4D input is supported. Padding should be constant, and is only
allowed for H and W dimensions.
* Relu
* ReluN1To1
* Relu6
* Reshape
* Only supported when target Core ML version is 2, not supported when
targeting Core ML 3.
* ResizeBilinear
* SoftMax
* Tanh
* TransposeConv
* Weights should be constant.
## Feedback
For issues, please create a
[GitHub](https://github.com/tensorflow/tensorflow/issues/new?template=50-other-issues.md)
issue with all the necessary details to reproduce.
## FAQ
* Does CoreML delegate support fallback to CPU if a graph contains unsupported
ops?
* Yes
* Does CoreML delegate work on iOS Simulator?
* Yes. The library includes x86 and x86_64 targets so it can run on
a simulator, but you will not see performance boost over CPU.
* Does TensorFlow Lite and CoreML delegate support MacOS?
* TensorFlow Lite is only tested on iOS but not MacOS.
* Is custom TF Lite ops supported?
* No, CoreML delegate does not support custom ops and they will fallback to
CPU.
## APIs
* [Core ML delegate Swift API](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/swift/Sources/CoreMLDelegate.swift)
* [Core ML delegate C API](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/delegates/coreml/coreml_delegate.h)
* This can be used for Objective-C codes. ~~~
tensorflow/lite/g3doc/performance/delegates.md
+263
View File
@@ -0,0 +1,263 @@
# TensorFlow Lite Delegates
<aside class="warning">
<p><b>Warning:</b> The
<a href="https://www.tensorflow.org/lite/android/delegates/nnapi">
NNAPI</a> and <a href="https://www.tensorflow.org/lite/android/delegates/hexagon">
Hexagon</a> delegates are deprecated and no longer supported by TensorFlow
Lite. For more information, see the
<a href="https://developer.android.com/ndk/guides/neuralnetworks/migration-guide">
NNAPI Migration Guide</a> and
<a href="https://www.tensorflow.org/lite/performance/delegates">TF Lite
delegates documentation</a>.</p>
</aside>
## Introduction
**Delegates** enable hardware acceleration of TensorFlow Lite models by
leveraging on-device accelerators such as the GPU and
[Digital Signal Processor (DSP)](https://en.wikipedia.org/wiki/Digital_signal_processor).
By default, TensorFlow Lite utilizes CPU kernels that are optimized for the
[ARM Neon](https://developer.arm.com/documentation/dht0002/a/Introducing-NEON/NEON-architecture-overview/NEON-instructions)
instruction set. However, the CPU is a multi-purpose processor that isn't
necessarily optimized for the heavy arithmetic typically found in Machine
Learning models (for example, the matrix math involved in convolution and dense
layers).
On the other hand, most modern mobile phones contain chips that are better at
handling these heavy operations. Utilizing them for neural network operations
provides huge benefits in terms of latency and power efficiency. For example,
GPUs can provide upto a
[5x speedup](https://blog.tensorflow.org/2020/08/faster-mobile-gpu-inference-with-opencl.html)
in latency, while the
[Qualcomm® Hexagon DSP](https://developer.qualcomm.com/software/hexagon-dsp-sdk/dsp-processor)
has shown to reduce power consumption upto 75% in our experiments.
Each of these accelerators have associated APIs that enable custom computations,
such as [OpenCL](https://www.khronos.org/opencl/) or
[OpenGL ES](https://www.khronos.org/opengles/) for mobile GPU and the
[Qualcomm® Hexagon SDK](https://developer.qualcomm.com/software/hexagon-dsp-sdk)
for DSP. Typically, you would have to write a lot of custom code to run a neural
network through these interfaces. Things get even more complicated when you
consider that each accelerator has its pros & cons and cannot execute every
operation in a neural network. TensorFlow Lite's Delegate API solves this
problem by acting as a bridge between the TFLite runtime and these lower-level
APIs.
![runtime with delegates](images/delegate_runtime.png)
## Choosing a Delegate
TensorFlow Lite supports multiple delegates, each of which is optimized for
certain platform(s) and particular types of models. Usually, there will be
multiple delegates applicable to your use-case, depending on two major criteria:
the *Platform* (Android or iOS?) you target, and the *Model-type*
(floating-point or quantized?) that you are trying to accelerate.
### Delegates by Platform
#### Cross-platform (Android & iOS)
* **GPU delegate** - The GPU delegate can be used on both Android and iOS. It
is optimized to run 32-bit and 16-bit float based models where a GPU is
available. It also supports 8-bit quantized models and provides GPU
performance on par with their float versions. For details on the GPU
delegate, see [TensorFlow Lite on GPU](gpu_advanced.md). For step-by-step
tutorials on using the GPU delegate with Android and iOS, see
[TensorFlow Lite GPU Delegate Tutorial](gpu.md).
#### Android
* **NNAPI delegate for newer Android devices** - The NNAPI delegate can be
used to accelerate models on Android devices with GPU, DSP and / or NPU
available. It is available in Android 8.1 (API 27+) or higher. For an
overview of the NNAPI delegate, step-by-step instructions and best
practices, see [TensorFlow Lite NNAPI delegate](nnapi.md).
* **Hexagon delegate for older Android devices** - The Hexagon delegate can be
used to accelerate models on Android devices with Qualcomm Hexagon DSP. It
can be used on devices running older versions of Android that do not support
NNAPI. See [TensorFlow Lite Hexagon delegate](hexagon_delegate.md) for more
detail.
#### iOS
* **Core ML delegate for newer iPhones and iPads** - For newer iPhones and
iPads where Neural Engine is available, you can use Core ML delegate to
accelerate inference for 32-bit or 16-bit floating-point models. Neural
Engine is available Apple mobile devices with A12 SoC or higher. For an
overview of the Core ML delegate and step-by-step instructions, see
[TensorFlow Lite Core ML delegate](coreml_delegate.md).
### Delegates by model type
Each accelerator is designed with a certain bit-width of data in mind. If you
provide a floating-point model to a delegate that only supports 8-bit quantized
operations (such as the [Hexagon delegate](hexagon_delegate.md)), it will reject
all its operations and the model will run entirely on the CPU. To avoid such
surprises, the table below provides an overview of delegate support based on
model type:
**Model Type** | **GPU** | **NNAPI** | **Hexagon** | **CoreML**
------------------------------------------------------------------------------------------------------- | ------- | --------- | ----------- | ----------
Floating-point (32 bit) | Yes | Yes | No | Yes
[Post-training float16 quantization](post_training_float16_quant.ipynb) | Yes | No | No | Yes
[Post-training dynamic range quantization](post_training_quant.ipynb) | Yes | Yes | No | No
[Post-training integer quantization](post_training_integer_quant.ipynb) | Yes | Yes | Yes | No
[Quantization-aware training](http://www.tensorflow.org/model_optimization/guide/quantization/training) | Yes | Yes | Yes | No
### Validating performance
The information in this section acts as a rough guideline for shortlisting the
delegates that could improve your application. However, it is important to note
that each delegate has a pre-defined set of operations it supports, and may
perform differently depending on the model and device; for example, the
[NNAPI delegate](nnapi.md) may choose to use Google's Edge-TPU on a Pixel phone
while utilizing a DSP on another device. Therefore, it is usually recommended
that you perform some benchmarking to gauge how useful a delegate is for your
needs. This also helps justify the binary size increase associated with
attaching a delegate to the TensorFlow Lite runtime.
TensorFlow Lite has extensive performance and accuracy-evaluation tooling that
can empower developers to be confident in using delegates in their application.
These tools are discussed in the next section.
## Tools for Evaluation
### Latency & memory footprint
TensorFlow Lite’s
[benchmark tool](https://www.tensorflow.org/lite/performance/measurement) can be
used with suitable parameters to estimate model performance, including average
inference latency, initialization overhead, memory footprint, etc. This tool
supports multiple flags to figure out the best delegate configuration for your
model. For instance, `--gpu_backend=gl` can be specified with `--use_gpu` to
measure GPU execution with OpenGL. The complete list of supported delegate
parameters is defined in the
[detailed documentation](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/tools/delegates/README.md#tflite-delegate-registrar).
Here’s an example run for a quantized model with GPU via `adb`:
```
adb shell /data/local/tmp/benchmark_model \
--graph=/data/local/tmp/mobilenet_v1_224_quant.tflite \
--use_gpu=true
```
You can download pre-built version of this tool for Android, 64-bit ARM
architecture
[here](https://storage.googleapis.com/tensorflow-nightly-public/prod/tensorflow/release/lite/tools/nightly/latest/android_aarch64_benchmark_model.apk)
([more details](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/tools/benchmark/android)).
### Accuracy & correctness
Delegates usually perform computations at a different precision than their CPU
counterparts. As a result, there is an (usually minor) accuracy tradeoff
associated with utilizing a delegate for hardware acceleration. Note that this
isn't *always* true; for example, since the GPU uses floating-point precision to
run quantized models, there might be a slight precision improvement (for e.g.,
<1% Top-5 improvement in ILSVRC image classification).
TensorFlow Lite has two types of tooling to measure how accurately a delegate
behaves for a given model: *Task-Based* and *Task-Agnostic*. All the tools
described in this section support the
[advanced delegation parameters](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/tools/delegates/README.md#tflite-delegate-registrar)
used by the benchmarking tool from the previous section. Note that the
sub-sections below focus on *delegate evaluation* (Does the delegate perform the
same as the CPU?) rather than model evaluation (Is the model itself good for the
task?).
#### Task-Based Evaluation
TensorFlow Lite has tools to evaluate correctness on two image-based tasks:
* [ILSVRC 2012](http://image-net.org/challenges/LSVRC/2012/) (Image
Classification) with
[top-K accuracy](https://en.wikipedia.org/wiki/Evaluation_measures_\(information_retrieval\)#Precision_at_K)
* [COCO Object Detection (w/ bounding boxes)](https://cocodataset.org/#detection-2020)
with
[mean Average Precision (mAP)](https://en.wikipedia.org/wiki/Evaluation_measures_\(information_retrieval\)#Mean_average_precision)
Prebuilt binaries of these tools (Android, 64-bit ARM architecture), along with
documentation can be found here:
* [ImageNet Image Classification](https://storage.googleapis.com/tensorflow-nightly-public/prod/tensorflow/release/lite/tools/nightly/latest/android_aarch64_eval_imagenet_image_classification)
([More details](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/tools/evaluation/tasks/imagenet_image_classification))
* [COCO Object Detection](https://storage.googleapis.com/tensorflow-nightly-public/prod/tensorflow/release/lite/tools/nightly/latest/android_aarch64_eval_coco_object_detection)
([More details](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/tools/evaluation/tasks/coco_object_detection))
The example below demonstrates
[image classification evaluation](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/tools/evaluation/tasks/imagenet_image_classification)
with NNAPI utilizing Google's Edge-TPU on a Pixel 4:
```
adb shell /data/local/tmp/run_eval \
--model_file=/data/local/tmp/mobilenet_quant_v1_224.tflite \
--ground_truth_images_path=/data/local/tmp/ilsvrc_images \
--ground_truth_labels=/data/local/tmp/ilsvrc_validation_labels.txt \
--model_output_labels=/data/local/tmp/model_output_labels.txt \
--output_file_path=/data/local/tmp/accuracy_output.txt \
--num_images=0 # Run on all images. \
--use_nnapi=true \
--nnapi_accelerator_name=google-edgetpu
```
The expected output is a list of Top-K metrics from 1 to 10:
```
Top-1 Accuracy: 0.733333
Top-2 Accuracy: 0.826667
Top-3 Accuracy: 0.856667
Top-4 Accuracy: 0.87
Top-5 Accuracy: 0.89
Top-6 Accuracy: 0.903333
Top-7 Accuracy: 0.906667
Top-8 Accuracy: 0.913333
Top-9 Accuracy: 0.92
Top-10 Accuracy: 0.923333
```
#### Task-Agnostic Evaluation
For tasks where there isn't an established on-device evaluation tool, or if you
are experimenting with custom models, TensorFlow Lite has the
[Inference Diff](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/tools/evaluation/tasks/inference_diff)
tool. (Android, 64-bit ARM binary architecture binary
[here](https://storage.googleapis.com/tensorflow-nightly-public/prod/tensorflow/release/lite/tools/nightly/latest/android_aarch64_eval_inference_diff))
Inference Diff compares TensorFlow Lite execution (in terms of latency &
output-value deviation) in two settings:
* Single-threaded CPU Inference
* User-defined Inference - defined by
[these parameters](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/tools/delegates/README.md#tflite-delegate-registrar)
To do so, the tool generates random Gaussian data and passes it through two
TFLite Interpreters - one running single-threaded CPU kernels, and the other
parameterized by the user's arguments.
It measures the latency of both, as well as the absolute difference between the
output tensors from each Interpreter, on a per-element basis.
For a model with a single output tensor, the output might look like this:
```
Num evaluation runs: 50
Reference run latency: avg=84364.2(us), std_dev=12525(us)
Test run latency: avg=7281.64(us), std_dev=2089(us)
OutputDiff[0]: avg_error=1.96277e-05, std_dev=6.95767e-06
```
What this means is that for the output tensor at index `0`, the elements from
the CPU output different from the delegate output by an average of `1.96e-05`.
Note that interpreting these numbers requires deeper knowledge of the model, and
what each output tensor signifies. If its a simple regression that determines
some sort of score or embedding, the difference should be low (otherwise it's an
error with the delegate). However, outputs like the 'detection class' one from
SSD models is a little harder to interpret. For example, it might show a
difference using this tool, but that may not mean something really wrong with
the delegate: consider two (fake) classes: "TV (ID: 10)", "Monitor (ID:20)" - If
a delegate is slightly off the golden truth and shows monitor instead of TV, the
output diff for this tensor might be something as high as 20-10 = 10.
tensorflow/lite/g3doc/performance/gpu.md
+218
View File
@@ -0,0 +1,218 @@
# GPU delegates for TensorFlow Lite
Using graphics processing units (GPUs) to run your machine learning (ML) models
can dramatically improve the performance of your model and the user experience
of your ML-enabled applications. TensorFlow Lite enables the use of GPUs and
other specialized processors through hardware driver called
[*delegates*](./delegates). Enabling use of GPUs with your TensorFlow Lite ML
applications can provide the following benefits:
* **Speed** - GPUs are built for high throughput of massively parallel
workloads. This design makes them well-suited for deep neural nets, which
consist of a huge number of operators, each working on input tensors that
can be processed in parallel, which typically results in lower latency. In
the best scenario, running your model on a GPU may run fast enough to enable
real-time applications that were not previously possible.
* **Power efficiency** - GPUs carry out ML computations in a very efficient
and optimized manner, typically consuming less power and generating less
heat than the same task running on CPUs.
This document provides an overview of GPUs support in TensorFlow Lite, and some
advanced uses for GPU processors. For more specific information about
implementing GPU support on specific platforms, see the following guides:
* [GPU support for Android](https://ai.google.dev/edge/litert/android/gpu)
* [GPU support for iOS](https://ai.google.dev/edge/litert/ios/gpu)
## GPU ML operations support {:#supported_ops}
There are some limitations to what TensorFlow ML operations, or *ops*, can be
accelerated by the TensorFlow Lite GPU delegate. The delegate supports the
following ops in 16-bit and 32-bit float precision:
* `ADD`
* `AVERAGE_POOL_2D`
* `CONCATENATION`
* `CONV_2D`
* `DEPTHWISE_CONV_2D v1-2`
* `EXP`
* `FULLY_CONNECTED`
* `LOGICAL_AND`
* `LOGISTIC`
* `LSTM v2 (Basic LSTM only)`
* `MAX_POOL_2D`
* `MAXIMUM`
* `MINIMUM`
* `MUL`
* `PAD`
* `PRELU`
* `RELU`
* `RELU6`
* `RESHAPE`
* `RESIZE_BILINEAR v1-3`
* `SOFTMAX`
* `STRIDED_SLICE`
* `SUB`
* `TRANSPOSE_CONV`
By default, all ops are only supported at version 1. Enabling the
[quantization support](#quantized-models) enables the appropriate versions, for
example, ADD v2.
### Troubleshooting GPU support
If some of the ops are not supported by the GPU delegate, the framework will
only run a part of the graph on the GPU and the remaining part on the CPU. Due
to the high cost of CPU/GPU synchronization, a split execution mode like this
often results in slower performance than when the whole network is run on
the CPU alone. In this case, the application generates warning, such as:
```none
WARNING: op code #42 cannot be handled by this delegate.
```
There is no callback for failures of this type, since this is not an actual
run-time failure. When testing execution of your model with the GPU delegate,
you should be alert for these warnings. A high number of these warnings can
indicate that your model is not the best fit for use for GPU acceleration, and
may require refactoring of the model.
## Example models
The following example models are built to take advantage GPU acceleration with
TensorFlow Lite and are provided for reference and testing:
* [MobileNet v1 (224x224) image classification](https://ai.googleblog.com/2017/06/mobilenets-open-source-models-for.html) -
An image classification model designed for mobile and embedded based vision
applications.
([model](https://tfhub.dev/google/imagenet/mobilenet_v1_100_224/classification/5))
* [DeepLab segmentation (257x257)](https://ai.googleblog.com/2018/03/semantic-image-segmentation-with.html) -
image segmentation model that assigns semantic labels, such as a dog, cat,
car, to every pixel in the input image.
([model](https://tfhub.dev/tensorflow/lite-model/deeplabv3/1/default/1))
* [MobileNet SSD object detection](https://ai.googleblog.com/2018/07/accelerated-training-and-inference-with.html) -
An image classification model that detects multiple objects with bounding
boxes.
([model](https://storage.googleapis.com/download.tensorflow.org/models/tflite/gpu/mobile_ssd_v2_float_coco.tflite))
* [PoseNet for pose estimation](https://github.com/tensorflow/tfjs-models/tree/master/pose-detection) -
A vision model that estimates the poses of people in image or video.
([model](https://tfhub.dev/tensorflow/lite-model/posenet/mobilenet/float/075/1/default/1))
## Optimizing for GPUs
The following techniques can help you get better performance when running
models on GPU hardware using the TensorFlow Lite GPU delegate:
* **Reshape operations** - Some operations that are quick on a CPU may have a
high cost for the GPU on mobile devices. Reshape operations are particularly
expensive to run, including `BATCH_TO_SPACE`, `SPACE_TO_BATCH`,
`SPACE_TO_DEPTH`, and so forth. You should closely examine use of reshape
operations, and consider that may have been applied only for exploring data
or for early iterations of your model. Removing them can significantly
improve performance.
* **Image data channels** - On GPU, tensor data is sliced into 4-channels, and
so a computation on a tensor with the shape `[B,H,W,5]` performs about the
same on a tensor of shape `[B,H,W,8]`, but significantly worse than
`[B,H,W,4]`. If the camera hardware you are using supports image frames in
RGBA, feeding that 4-channel input is significantly faster, since it avoids
a memory copy from 3-channel RGB to 4-channel RGBX.
* **Mobile-optimized models** - For best performance, you should consider
retraining your classifier with a mobile-optimized network architecture.
Optimization for on-device inferencing can dramatically reduce latency and
power consumption by taking advantage of mobile hardware features.
## Advanced GPU support
You can use additional, advanced techniques with GPU processing to enable even
better performance for your models, including quantization and serialization.
The following sections describe these techniques in further detail.
### Using quantized models {:#quantized-models}
This section explains how the GPU delegate accelerates 8-bit quantized models,
including the following:
* Models trained with
[Quantization-aware training](https://www.tensorflow.org/model_optimization/guide/quantization/training)
* Post-training [dynamic-range quantization](https://www.tensorflow.org/lite/performance/post_training_quant)
* Post-training [full-integer quantization](https://www.tensorflow.org/lite/performance/post_training_integer_quant)
To optimize performance, use models that have both floating-point input and
output tensors.
#### How does this work?
Since the GPU backend only supports floating-point execution, we run quantized
models by giving it a ‘floating-point view’ of the original model. At a
high-level, this entails the following steps:
* *Constant tensors* (such as weights/biases) are de-quantized once into the
GPU memory. This operation happens when the delegate is enabled for
TensorFlow Lite.
* *Inputs and outputs* to the GPU program, if 8-bit quantized, are
de-quantized and quantized (respectively) for each inference. This operation
is done on the CPU using TensorFlow Lite’s optimized kernels.
* *Quantization simulators* are inserted between operations to mimic quantized
behavior. This approach is necessary for models where ops expect activations
to follow bounds learnt during quantization.
For information about enabling this feature with the GPU delegate, see the
following:
* Using [quantized models with GPU on Android](../android/delegates/gpu#quantized-models)
* Using [quantized models with GPU on iOS](../ios/delegates/gpu#quantized-models)
### Reducing initialization time with serialization {:#delegate_serialization}
The GPU delegate feature allows you to load from pre-compiled kernel code and
model data serialized and saved on disk from previous runs. This approach avoids
re-compilation and can reduce startup time by up to 90%. This improvement is
achieved by exchanging disk space for time savings. You can enable this feature
with a few configurations options, as shown in the following code examples:
<div>
<devsite-selector>
<section>
<h3>C++</h3>
<p><pre class="prettyprint lang-cpp">
TfLiteGpuDelegateOptionsV2 options = TfLiteGpuDelegateOptionsV2Default();
options.experimental_flags |= TFLITE_GPU_EXPERIMENTAL_FLAGS_ENABLE_SERIALIZATION;
options.serialization_dir = kTmpDir;
options.model_token = kModelToken;
auto* delegate = TfLiteGpuDelegateV2Create(options);
if (interpreter->ModifyGraphWithDelegate(delegate) != kTfLiteOk) return false;
</pre></p>
</section>
<section>
<h3>Java</h3>
<p><pre class="prettyprint lang-java">
GpuDelegate delegate = new GpuDelegate(
new GpuDelegate.Options().setSerializationParams(
/* serializationDir= */ serializationDir,
/* modelToken= */ modelToken));
Interpreter.Options options = (new Interpreter.Options()).addDelegate(delegate);
</pre></p>
</section>
</devsite-selector>
</div>
When using the serialization feature, make sure your code complies with these
implementation rules:
* Store the serialization data in a directory that is not accessible to other
apps. On Android devices, use
[`getCodeCacheDir()`](https://developer.android.com/reference/android/content/Context#getCacheDir\(\))
which points to a location that is private to the current application.
* The model token must be unique to the device for the specific model. You can
compute a model token by generating a fingerprint from the model data
using libraries such as
[`farmhash::Fingerprint64`](https://github.com/google/farmhash).
Note: Use of this serialization feature requires the
[OpenCL SDK](https://github.com/KhronosGroup/OpenCL-SDK).
tensorflow/lite/g3doc/performance/images/android_gpu_demo.gif
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.4 MiB

tensorflow/lite/g3doc/performance/images/as_select_profiling_mode.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 144 KiB

tensorflow/lite/g3doc/performance/images/as_traces.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 230 KiB

tensorflow/lite/g3doc/performance/images/delegate_runtime.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 34 KiB

tensorflow/lite/g3doc/performance/images/optimization.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 82 KiB

tensorflow/lite/g3doc/performance/images/perfetto_traces.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 124 KiB

tensorflow/lite/g3doc/performance/images/quantization_decision_tree.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 286 KiB

tensorflow/lite/g3doc/performance/images/tflite_delegate.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 164 KiB

tensorflow/lite/g3doc/performance/images/xcode_profile_environment.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 154 KiB

tensorflow/lite/g3doc/performance/images/xcode_traces.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 314 KiB

tensorflow/lite/g3doc/performance/implementing_delegate.md
+618
View File
@@ -0,0 +1,618 @@
# Implementing a Custom Delegate
<aside class="warning">
<p><b>Warning:</b> The
<a href="https://www.tensorflow.org/lite/android/delegates/nnapi">
NNAPI</a> and <a href="https://www.tensorflow.org/lite/android/delegates/hexagon">
Hexagon</a> delegates are deprecated and no longer supported by TensorFlow
Lite. For more information, see the
<a href="https://developer.android.com/ndk/guides/neuralnetworks/migration-guide">
NNAPI Migration Guide</a> and
<a href="https://www.tensorflow.org/lite/performance/delegates">TF Lite
delegates documentation</a>.</p>
</aside>
[TOC]
## What is a TensorFlow Lite Delegate?
A TensorFlow Lite
[Delegate](https://www.tensorflow.org/lite/performance/delegates) allows you to
run your models (part or whole) on another executor. This mechanism can leverage
a variety of on-device accelerators such as the GPU or Edge TPU (Tensor
Processing Unit) for inference. This provides developers a flexible and
decoupled method from the default TFLite to speed up inference.
Diagram below summarizes the delegates, more details in the below sections.
![TFLite Delegates](images/tflite_delegate.png "TFLite Delegates")
## When should I create a Custom delegate?
TensorFlow Lite has a wide variety of delegates for target accelerators such as
GPU, DSP, EdgeTPU and frameworks like Android NNAPI.
Creating your own delegate is useful in the following scenarios:
* You want to integrate a new ML inference engine not supported by any
existing delegate.
* You have a custom hardware accelerator that improves runtime for known
scenarios.
* You are developing CPU optimizations (such as operator fusing) that can
speed up certain models.
## How do delegates work?
Consider a simple model graph such as the following, and a delegate “MyDelegate”
that has a faster implementation for Conv2D and Mean operations.
![Original graph](../images/performance/tflite_delegate_graph_1.png "Original Graph")
After applying this “MyDelegate”, the original TensorFlow Lite graph will be
updated like the following:
![Graph with delegate](../images/performance/tflite_delegate_graph_2.png "Graph with delegate")
The graph above is obtained as TensorFlow Lite splits the original graph
following two rules:
* Specific operations that could be handled by the delegate are put into a
partition while still satisfying the original computing workflow
dependencies among operations.
* Each to-be-delegated partition only has input and output nodes that are not
handled by the delegate.
Each partition that is handled by a delegate is replaced by a delegate node (can
also be called as a delegate kernel) in the original graph that evaluates the
partition on its invoke call.
Depending on the model, the final graph can end up with one or more nodes, the
latter meaning that some ops are not supported by the delegate. In general, you
don’t want to have multiple partitions handled by the delegate, because each
time you switch from delegate to the main graph, there is an overhead for
passing the results from the delegated subgraph to the main graph that results
due to memory copies (for example, GPU to CPU). Such overhead might offset
performance gains especially when there are a large amount of memory copies.
## Implementing your own Custom delegate
The preferred method to add a delegate is using
[SimpleDelegate API](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/delegates/utils/simple_delegate.h).
To create a new delegate, you need to implement 2 interfaces and provide your
own implementation for the interface methods.
### 1 - `SimpleDelegateInterface`
This class represents the capabilities of the delegate, which operations are
supported, and a factory class for creating a kernel which encapsulates the
delegated graph. For more details, see the interface defined in this
[C++ header file](https://github.com/tensorflow/tensorflow/blob/8a643858ce174b8bd1b4bb8fa4bfaa62f7e8c45f/tensorflow/lite/delegates/utils/simple_delegate.h#L71).
The comments in the code explain each API in detail.
### 2 - `SimpleDelegateKernelInterface`
This class encapsulates the logic for initializing / preparing / and running the
delegated partition.
It has: (See
[definition](https://github.com/tensorflow/tensorflow/blob/8a643858ce174b8bd1b4bb8fa4bfaa62f7e8c45f/tensorflow/lite/delegates/utils/simple_delegate.h#L43))
* Init(...): which will be called once to do any one-time initialization.
* Prepare(...): called for each different instance of this node - this happens
if you have multiple delegated partitions. Usually you want to do memory
allocations here, since this will be called everytime tensors are resized.
* Invoke(...): which will be called for inference.
### Example
In this example, you will create a very simple delegate that can support only 2
types of operations (ADD) and (SUB) with float32 tensors only.
```
// MyDelegate implements the interface of SimpleDelegateInterface.
// This holds the Delegate capabilities.
class MyDelegate : public SimpleDelegateInterface {
public:
bool IsNodeSupportedByDelegate(const TfLiteRegistration* registration,
const TfLiteNode* node,
TfLiteContext* context) const override {
// Only supports Add and Sub ops.
if (kTfLiteBuiltinAdd != registration->builtin_code &&
kTfLiteBuiltinSub != registration->builtin_code)
return false;
// This delegate only supports float32 types.
for (int i = 0; i < node->inputs->size; ++i) {
auto& tensor = context->tensors[node->inputs->data[i]];
if (tensor.type != kTfLiteFloat32) return false;
}
return true;
}
TfLiteStatus Initialize(TfLiteContext* context) override { return kTfLiteOk; }
const char* Name() const override {
static constexpr char kName[] = "MyDelegate";
return kName;
}
std::unique_ptr<SimpleDelegateKernelInterface> CreateDelegateKernelInterface()
override {
return std::make_unique<MyDelegateKernel>();
}
};
```
Next, create your own delegate kernel by inheriting from the
`SimpleDelegateKernelInterface`
```
// My delegate kernel.
class MyDelegateKernel : public SimpleDelegateKernelInterface {
public:
TfLiteStatus Init(TfLiteContext* context,
const TfLiteDelegateParams* params) override {
// Save index to all nodes which are part of this delegate.
inputs_.resize(params->nodes_to_replace->size);
outputs_.resize(params->nodes_to_replace->size);
builtin_code_.resize(params->nodes_to_replace->size);
for (int i = 0; i < params->nodes_to_replace->size; ++i) {
const int node_index = params->nodes_to_replace->data[i];
// Get this node information.
TfLiteNode* delegated_node = nullptr;
TfLiteRegistration* delegated_node_registration = nullptr;
TF_LITE_ENSURE_EQ(
context,
context->GetNodeAndRegistration(context, node_index, &delegated_node,
&delegated_node_registration),
kTfLiteOk);
inputs_[i].push_back(delegated_node->inputs->data[0]);
inputs_[i].push_back(delegated_node->inputs->data[1]);
outputs_[i].push_back(delegated_node->outputs->data[0]);
builtin_code_[i] = delegated_node_registration->builtin_code;
}
return kTfLiteOk;
}
TfLiteStatus Prepare(TfLiteContext* context, TfLiteNode* node) override {
return kTfLiteOk;
}
TfLiteStatus Eval(TfLiteContext* context, TfLiteNode* node) override {
// Evaluate the delegated graph.
// Here we loop over all the delegated nodes.
// We know that all the nodes are either ADD or SUB operations and the
// number of nodes equals ''inputs_.size()'' and inputs[i] is a list of
// tensor indices for inputs to node ''i'', while outputs_[i] is the list of
// outputs for node
// ''i''. Note, that it is intentional we have simple implementation as this
// is for demonstration.
for (int i = 0; i < inputs_.size(); ++i) {
// Get the node input tensors.
// Add/Sub operation accepts 2 inputs.
auto& input_tensor_1 = context->tensors[inputs_[i][0]];
auto& input_tensor_2 = context->tensors[inputs_[i][1]];
auto& output_tensor = context->tensors[outputs_[i][0]];
TF_LITE_ENSURE_EQ(
context,
ComputeResult(context, builtin_code_[i], &input_tensor_1,
&input_tensor_2, &output_tensor),
kTfLiteOk);
}
return kTfLiteOk;
}
private:
// Computes the result of addition of 'input_tensor_1' and 'input_tensor_2'
// and store the result in 'output_tensor'.
TfLiteStatus ComputeResult(TfLiteContext* context, int builtin_code,
const TfLiteTensor* input_tensor_1,
const TfLiteTensor* input_tensor_2,
TfLiteTensor* output_tensor) {
if (NumElements(input_tensor_1) != NumElements(input_tensor_2) ||
NumElements(input_tensor_1) != NumElements(output_tensor)) {
return kTfLiteDelegateError;
}
// This code assumes no activation, and no broadcasting needed (both inputs
// have the same size).
auto* input_1 = GetTensorData<float>(input_tensor_1);
auto* input_2 = GetTensorData<float>(input_tensor_2);
auto* output = GetTensorData<float>(output_tensor);
for (int i = 0; i < NumElements(input_tensor_1); ++i) {
if (builtin_code == kTfLiteBuiltinAdd)
output[i] = input_1[i] + input_2[i];
else
output[i] = input_1[i] - input_2[i];
}
return kTfLiteOk;
}
// Holds the indices of the input/output tensors.
// inputs_[i] is list of all input tensors to node at index 'i'.
// outputs_[i] is list of all output tensors to node at index 'i'.
std::vector<std::vector<int>> inputs_, outputs_;
// Holds the builtin code of the ops.
// builtin_code_[i] is the type of node at index 'i'
std::vector<int> builtin_code_;
};
```
## Benchmark and evaluate the new delegate
TFLite has a set of tools that you can quickly test against a TFLite model.
* [Model Benchmark Tool](https://github.com/tensorflow/tensorflow/tree/f9ef3a8a0b64ad6393785f3259e9a24af09c84ad/tensorflow/lite/tools/benchmark):
The tool takes a TFLite model, generates random inputs, and then repeatedly
runs the model for a specified number of runs. It prints aggregated latency
statistics at the end.
* [Inference Diff Tool](https://github.com/tensorflow/tensorflow/tree/f9ef3a8a0b64ad6393785f3259e9a24af09c84ad/tensorflow/lite/tools/evaluation/tasks/inference_diff):
For a given model, the tool generates random Gaussian data and passes it
through two different TFLite interpreters, one running single threaded CPU
kernel and the other using a user-defined spec. It measures the absolute
difference between the output tensors from each interpreter, on a
per-element basis. This tool can also be helpful for debugging accuracy
issues.
* There are also task specific evaluation tools, for image classification and
object detection. These tools can be found
[here](https://www.tensorflow.org/lite/performance/delegates#tools_for_evaluation)
In addition, TFLite has a large set of kernel and op unit tests that could be
reused to test the new delegate with more coverage and to ensure the regular
TFLite execution path is not broken.
To achieve reusing TFLite tests and tooling for the new delegate, you can use
either of the following two options:
* Utilize the
[delegate registrar](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/tools/delegates)
mechanism.
* Utilize the
[external delegate](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/delegates/external)
mechanism.
### Choosing the best approach
Both approaches require a few changes as detailed below. However, the first
approach links the delegate statically and requires rebuilding the testing,
benchmarking and evaluation tools. In contrast, the second one makes the
delegate as a shared library and requires you to expose the create/delete
methods from the shared library.
As a result, the external-delegate mechanism will work with TFLite’s
[pre-built Tensorflow Lite tooling binaries](#download-links-for-nightly-pre-built-tflite-tooling-binaries).
But it is less explicit and it might be more complicated to set up in automated
integration tests. Use the delegate registrar approach for better clarity.
### Option 1: Leverage delegate registrar
The
[delegate registrar](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/tools/delegates)
keeps a list of delegate providers, each of which provides an easy way to create
TFLite delegates based on command-line flags, and are hence, convenient for
tooling. To plug in the new delegate to all the Tensorflow Lite tools mentioned
above, you first create a new delegate provider like this
[one](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/tools/delegates/hexagon_delegate_provider.cc),
and then makes only a few changes to the BUILD rules. A full example of this
integration process is shown below (and code can be found
[here](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/delegates/utils/dummy_delegate)).
Assuming you have a delegate that implements the SimpleDelegate APIs, and the
extern "C" APIs of creating/deleting this 'dummy' delegate as shown below:
```
// Returns default options for DummyDelegate.
DummyDelegateOptions TfLiteDummyDelegateOptionsDefault();
// Creates a new delegate instance that need to be destroyed with
// `TfLiteDummyDelegateDelete` when delegate is no longer used by TFLite.
// When `options` is set to `nullptr`, the above default values are used:
TfLiteDelegate* TfLiteDummyDelegateCreate(const DummyDelegateOptions* options);
// Destroys a delegate created with `TfLiteDummyDelegateCreate` call.
void TfLiteDummyDelegateDelete(TfLiteDelegate* delegate);
```
To integrate the “DummyDelegate” with Benchmark Tool and Inference Tool, define
a DelegateProvider like below:
```
class DummyDelegateProvider : public DelegateProvider {
public:
DummyDelegateProvider() {
default_params_.AddParam("use_dummy_delegate",
ToolParam::Create<bool>(false));
}
std::vector<Flag> CreateFlags(ToolParams* params) const final;
void LogParams(const ToolParams& params) const final;
TfLiteDelegatePtr CreateTfLiteDelegate(const ToolParams& params) const final;
std::string GetName() const final { return "DummyDelegate"; }
};
REGISTER_DELEGATE_PROVIDER(DummyDelegateProvider);
std::vector<Flag> DummyDelegateProvider::CreateFlags(ToolParams* params) const {
std::vector<Flag> flags = {CreateFlag<bool>("use_dummy_delegate", params,
"use the dummy delegate.")};
return flags;
}
void DummyDelegateProvider::LogParams(const ToolParams& params) const {
TFLITE_LOG(INFO) << "Use dummy test delegate : ["
<< params.Get<bool>("use_dummy_delegate") << "]";
}
TfLiteDelegatePtr DummyDelegateProvider::CreateTfLiteDelegate(
const ToolParams& params) const {
if (params.Get<bool>("use_dummy_delegate")) {
auto default_options = TfLiteDummyDelegateOptionsDefault();
return TfLiteDummyDelegateCreateUnique(&default_options);
}
return TfLiteDelegatePtr(nullptr, [](TfLiteDelegate*) {});
}
```
The BUILD rule definitions are important as you need to make sure that the
library is always linked and not dropped by optimizer.
```
#### The following are for using the dummy test delegate in TFLite tooling ####
cc_library(
name = "dummy_delegate_provider",
srcs = ["dummy_delegate_provider.cc"],
copts = tflite_copts(),
deps = [
":dummy_delegate",
"//tensorflow/lite/tools/delegates:delegate_provider_hdr",
],
alwayslink = 1, # This is required so the optimizer doesn't optimize the library away.
)
```
Now add these two wrapper rules in your BUILD file to create a version of
Benchmark Tool and Inference Tool, and other evaluation tools, that could run
with your own delegate.
```
cc_binary(
name = "benchmark_model_plus_dummy_delegate",
copts = tflite_copts(),
linkopts = task_linkopts(),
deps = [
":dummy_delegate_provider",
"//tensorflow/lite/tools/benchmark:benchmark_model_main",
],
)
cc_binary(
name = "inference_diff_plus_dummy_delegate",
copts = tflite_copts(),
linkopts = task_linkopts(),
deps = [
":dummy_delegate_provider",
"//tensorflow/lite/tools/evaluation/tasks:task_executor_main",
"//tensorflow/lite/tools/evaluation/tasks/inference_diff:run_eval_lib",
],
)
cc_binary(
name = "imagenet_classification_eval_plus_dummy_delegate",
copts = tflite_copts(),
linkopts = task_linkopts(),
deps = [
":dummy_delegate_provider",
"//tensorflow/lite/tools/evaluation/tasks:task_executor_main",
"//tensorflow/lite/tools/evaluation/tasks/imagenet_image_classification:run_eval_lib",
],
)
cc_binary(
name = "coco_object_detection_eval_plus_dummy_delegate",
copts = tflite_copts(),
linkopts = task_linkopts(),
deps = [
":dummy_delegate_provider",
"//tensorflow/lite/tools/evaluation/tasks:task_executor_main",
"//tensorflow/lite/tools/evaluation/tasks/coco_object_detection:run_eval_lib",
],
)
```
You can also plug in this delegate provider to TFLite kernel tests as described
[here](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/delegates/utils/dummy_delegate/README.md#kernel-tests).
### Option 2: Leverage external delegate
In this alternative, you first create an external delegate adaptor the
[external\_delegate\_adaptor.cc](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/delegates/utils/dummy_delegate/external_delegate_adaptor.cc)
as shown below. Note, this approach is slightly less preferred as compared to
Option 1 as has been [aforementioned](#comparison-between-the-two-options).
```
TfLiteDelegate* CreateDummyDelegateFromOptions(char** options_keys,
char** options_values,
size_t num_options) {
DummyDelegateOptions options = TfLiteDummyDelegateOptionsDefault();
// Parse key-values options to DummyDelegateOptions.
// You can achieve this by mimicking them as command-line flags.
std::unique_ptr<const char*> argv =
std::unique_ptr<const char*>(new const char*[num_options + 1]);
constexpr char kDummyDelegateParsing[] = "dummy_delegate_parsing";
argv.get()[0] = kDummyDelegateParsing;
std::vector<std::string> option_args;
option_args.reserve(num_options);
for (int i = 0; i < num_options; ++i) {
option_args.emplace_back("--");
option_args.rbegin()->append(options_keys[i]);
option_args.rbegin()->push_back('=');
option_args.rbegin()->append(options_values[i]);
argv.get()[i + 1] = option_args.rbegin()->c_str();
}
// Define command-line flags.
// ...
std::vector<tflite::Flag> flag_list = {
tflite::Flag::CreateFlag(...),
...,
tflite::Flag::CreateFlag(...),
};
int argc = num_options + 1;
if (!tflite::Flags::Parse(&argc, argv.get(), flag_list)) {
return nullptr;
}
return TfLiteDummyDelegateCreate(&options);
}
#ifdef __cplusplus
extern "C" {
#endif // __cplusplus
// Defines two symbols that need to be exported to use the TFLite external
// delegate. See tensorflow/lite/delegates/external for details.
TFL_CAPI_EXPORT TfLiteDelegate* tflite_plugin_create_delegate(
char** options_keys, char** options_values, size_t num_options,
void (*report_error)(const char*)) {
return tflite::tools::CreateDummyDelegateFromOptions(
options_keys, options_values, num_options);
}
TFL_CAPI_EXPORT void tflite_plugin_destroy_delegate(TfLiteDelegate* delegate) {
TfLiteDummyDelegateDelete(delegate);
}
#ifdef __cplusplus
}
#endif // __cplusplus
```
Now create the corresponding BUILD target to build a dynamic library as shown
below:
```
cc_binary(
name = "dummy_external_delegate.so",
srcs = [
"external_delegate_adaptor.cc",
],
linkshared = 1,
linkstatic = 1,
deps = [
":dummy_delegate",
"//tensorflow/lite/c:common",
"//tensorflow/lite/tools:command_line_flags",
"//tensorflow/lite/tools:logging",
],
)
```
After this external delegate .so file is created, you can build binaries or use
pre-built ones to run with the new delegate as long as the binary is linked with
the
[external\_delegate\_provider](https://github.com/tensorflow/tensorflow/blob/8c6f2d55762f3fc94f98fdd8b3c5d59ee1276dba/tensorflow/lite/tools/delegates/BUILD#L145-L159)
library which supports command-line flags as described
[here](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/tools/delegates#external-delegate-provider).
Note: this external delegate provider has already been linked to existing
testing and tooling binaries.
Refer to descriptions
[here](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/delegates/utils/dummy_delegate/README.md#option-2-utilize-tensorflow-lite-external-delegate)
for an illustration of how to benchmark the dummy delegate via this
external-delegate approach. You can use similar commands for the testing and
evaluation tools mentioned earlier.
It is worth noting the _external delegate_ is the corresponding C++
implementation of the _delegate_ in Tensorflow Lite Python binding as shown
[here](https://github.com/tensorflow/tensorflow/blob/7145fc0e49be01ef6943f4df386ce38567e37797/tensorflow/lite/python/interpreter.py#L42).
Therefore, the dynamic external delegate adaptor library created here could be
directly used with Tensorflow Lite Python APIs.
## Resources
### Download links for nightly pre-built TFLite tooling binaries
<table>
<tr>
<td>OS
</td>
<td>ARCH
</td>
<td>BINARY_NAME
</td>
</tr>
<tr>
<td rowspan="3" >Linux
</td>
<td>x86_64
</td>
<td><ul>
<li><a href="https://storage.googleapis.com/tensorflow-nightly-public/prod/tensorflow/release/lite/tools/nightly/latest/linux_x86-64_benchmark_model">benchmark_model</a>
<li><a href="https://storage.googleapis.com/tensorflow-nightly-public/prod/tensorflow/release/lite/tools/nightly/latest/linux_x86-64_eval_inference_diff">inference_diff</a>
<li><a href="https://storage.googleapis.com/tensorflow-nightly-public/prod/tensorflow/release/lite/tools/nightly/latest/linux_x86-64_eval_imagenet_image_classification">imagenet_image_classification_eval</a>
<li><a href="https://storage.googleapis.com/tensorflow-nightly-public/prod/tensorflow/release/lite/tools/nightly/latest/linux_x86-64_eval_coco_object_detection">coco_object_detection_eval</a></li></ul>
</td>
</tr>
<tr>
<td>arm
</td>
<td><ul>
<li><a href="https://storage.googleapis.com/tensorflow-nightly-public/prod/tensorflow/release/lite/tools/nightly/latest/linux_arm_benchmark_model">benchmark_model</a>
<li><a href="https://storage.googleapis.com/tensorflow-nightly-public/prod/tensorflow/release/lite/tools/nightly/latest/linux_arm_eval_inference_diff">inference_diff</a>
<li><a href="https://storage.googleapis.com/tensorflow-nightly-public/prod/tensorflow/release/lite/tools/nightly/latest/linux_arm_eval_imagenet_image_classification">imagenet_image_classification_eval</a>
<li><a href="https://storage.googleapis.com/tensorflow-nightly-public/prod/tensorflow/release/lite/tools/nightly/latest/linux_arm_eval_coco_object_detection">coco_object_detection_eval</a></li></ul>
</td>
</tr>
<tr>
<td>aarch64
</td>
<td><ul>
<li><a href="https://storage.googleapis.com/tensorflow-nightly-public/prod/tensorflow/release/lite/tools/nightly/latest/linux_aarch64_benchmark_model">benchmark_model</a>
<li><a href="https://storage.googleapis.com/tensorflow-nightly-public/prod/tensorflow/release/lite/tools/nightly/latest/linux_aarch64_eval_inference_diff">inference_diff</a>
<li><a href="https://storage.googleapis.com/tensorflow-nightly-public/prod/tensorflow/release/lite/tools/nightly/latest/linux_aarch64_eval_imagenet_image_classification">imagenet_image_classification_eval</a>
<li><a href="https://storage.googleapis.com/tensorflow-nightly-public/prod/tensorflow/release/lite/tools/nightly/latest/linux_aarch64_eval_coco_object_detection">coco_object_detection_eval</a></li></ul>
</td>
</tr>
<tr>
<td rowspan="2" >Android
</td>
<td>arm
</td>
<td><ul>
<li><a href="http://storage.googleapis.com/tensorflow-nightly-public/prod/tensorflow/release/lite/tools/nightly/latest/android_arm_benchmark_model">benchmark_model</a>
<li><strong><a href="https://storage.googleapis.com/tensorflow-nightly-public/prod/tensorflow/release/lite/tools/nightly/latest/android_arm_benchmark_model.apk">benchmark_model.apk</a></strong>
<li><a href="https://storage.googleapis.com/tensorflow-nightly-public/prod/tensorflow/release/lite/tools/nightly/latest/android_arm_eval_inference_diff">inference_diff</a>
<li><a href="https://storage.googleapis.com/tensorflow-nightly-public/prod/tensorflow/release/lite/tools/nightly/latest/android_arm_eval_imagenet_image_classification">imagenet_image_classification_eval</a>
<li><a href="https://storage.googleapis.com/tensorflow-nightly-public/prod/tensorflow/release/lite/tools/nightly/latest/android_arm_eval_coco_object_detection">coco_object_detection_eval</a></li></ul>
</td>
</tr>
<tr>
<td>aarch64
</td>
<td><ul>
<li><a href="http://storage.googleapis.com/tensorflow-nightly-public/prod/tensorflow/release/lite/tools/nightly/latest/android_aarch64_benchmark_model">benchmark_model</a>
<li><strong><a href="https://storage.googleapis.com/tensorflow-nightly-public/prod/tensorflow/release/lite/tools/nightly/latest/android_aarch64_benchmark_model.apk">benchmark_model.apk</a></strong>
<li><a href="https://storage.googleapis.com/tensorflow-nightly-public/prod/tensorflow/release/lite/tools/nightly/latest/android_aarch64_eval_inference_diff">inference_diff</a>
<li><a href="https://storage.googleapis.com/tensorflow-nightly-public/prod/tensorflow/release/lite/tools/nightly/latest/android_aarch64_eval_imagenet_image_classification">imagenet_image_classification_eval</a>
<li><a href="https://storage.googleapis.com/tensorflow-nightly-public/prod/tensorflow/release/lite/tools/nightly/latest/android_aarch64_eval_coco_object_detection">coco_object_detection_eval</a></li></ul>
</td>
</tr>
</table>
tensorflow/lite/g3doc/performance/measurement.md
+608
View File
@@ -0,0 +1,608 @@
# Performance measurement
## Benchmark tools
TensorFlow Lite benchmark tools currently measure and calculate statistics for
the following important performance metrics:
* Initialization time
* Inference time of warmup state
* Inference time of steady state
* Memory usage during initialization time
* Overall memory usage
The benchmark tools are available as benchmark apps for Android and iOS and as
native command-line binaries, and they all share the same core performance
measurement logic. Note that the available options and output formats are
slightly different due to the differences in runtime environment.
### Android benchmark app
There are two options of using the benchmark tool with Android. One is a
[native benchmark binary](#native-benchmark-binary) and another is an Android
benchmark app, a better gauge of how the model would perform in the app. Either
way, the numbers from the benchmark tool will still differ slightly from when
running inference with the model in the actual app.
This Android benchmark app has no UI. Install and run it by using the `adb`
command and retrieve results by using the `adb logcat` command.
#### Download or build the app
Download the nightly pre-built Android benchmark apps using the links below:
* [android_aarch64](https://storage.googleapis.com/tensorflow-nightly-public/prod/tensorflow/release/lite/tools/nightly/latest/android_aarch64_benchmark_model.apk)
* [android_arm](https://storage.googleapis.com/tensorflow-nightly-public/prod/tensorflow/release/lite/tools/nightly/latest/android_arm_benchmark_model.apk)
As for Android benchmark apps that support [TF ops](https://www.tensorflow.org/lite/guide/ops_select)
via [Flex delegate](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/delegates/flex),
use the links below:
* [android_aarch64](https://storage.googleapis.com/tensorflow-nightly-public/prod/tensorflow/release/lite/tools/nightly/latest/android_aarch64_benchmark_model_plus_flex.apk)
* [android_arm](https://storage.googleapis.com/tensorflow-nightly-public/prod/tensorflow/release/lite/tools/nightly/latest/android_arm_benchmark_model_plus_flex.apk)
You can also build the app from source by following these
[instructions](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/tools/benchmark/android).
Note: It is required to build the app from the source if you want to run the
Android benchmark apk on x86 CPU or Hexagon delegate or if your model contains
[select TF operators](../guide/ops_select) or
[custom operators](../guide/ops_custom).
#### Prepare benchmark
Before running the benchmark app, install the app and push the model file to the
device as follows:
```shell
adb install -r -d -g android_aarch64_benchmark_model.apk
adb push your_model.tflite /data/local/tmp
```
#### Run benchmark
```shell
adb shell am start -S \
-n org.tensorflow.lite.benchmark/.BenchmarkModelActivity \
--es args '"--graph=/data/local/tmp/your_model.tflite \
--num_threads=4"'
```
`graph` is a required parameter.
* `graph`: `string` \
The path to the TFLite model file.
You can specify more optional parameters for running the benchmark.
* `num_threads`: `int` (default=1) \
The number of threads to use for running TFLite interpreter.
* `use_gpu`: `bool` (default=false) \
Use [GPU delegate](gpu).
* `use_nnapi`: `bool` (default=false) \
Use [NNAPI delegate](nnapi).
* `use_xnnpack`: `bool` (default=`false`) \
Use
[XNNPACK delegate](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/delegates/xnnpack).
* `use_hexagon`: `bool` (default=`false`) \
Use [Hexagon delegate](hexagon_delegate).
Depending on the device you are using, some of these options may not be
available or have no effect. Refer to
[parameters](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/tools/benchmark#parameters)
for more performance parameters that you could run with the benchmark app.
View the results using the `logcat` command:
```shell
adb logcat | grep "Inference timings"
```
The benchmark results are reported as:
```
... tflite : Inference timings in us: Init: 5685, First inference: 18535, Warmup (avg): 14462.3, Inference (avg): 14575.2
```
### Native benchmark binary
Benchmark tool is also provided as a native binary `benchmark_model`. You can
execute this tool from a shell command line on Linux, Mac, embedded devices and
Android devices.
#### Download or build the binary
Download the nightly pre-built native command-line binaries by following the
links below:
* [linux_x86-64](https://storage.googleapis.com/tensorflow-nightly-public/prod/tensorflow/release/lite/tools/nightly/latest/linux_x86-64_benchmark_model)
* [linux_aarch64](https://storage.googleapis.com/tensorflow-nightly-public/prod/tensorflow/release/lite/tools/nightly/latest/linux_aarch64_benchmark_model)
* [linux_arm](https://storage.googleapis.com/tensorflow-nightly-public/prod/tensorflow/release/lite/tools/nightly/latest/linux_arm_benchmark_model)
* [android_aarch64](https://storage.googleapis.com/tensorflow-nightly-public/prod/tensorflow/release/lite/tools/nightly/latest/android_aarch64_benchmark_model)
* [android_arm](https://storage.googleapis.com/tensorflow-nightly-public/prod/tensorflow/release/lite/tools/nightly/latest/android_arm_benchmark_model)
As for nightly pre-built binaries that support [TF ops](https://www.tensorflow.org/lite/guide/ops_select)
via [Flex delegate](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/delegates/flex),
use the links below:
* [linux_x86-64](https://storage.googleapis.com/tensorflow-nightly-public/prod/tensorflow/release/lite/tools/nightly/latest/linux_x86-64_benchmark_model_plus_flex)
* [linux_aarch64](https://storage.googleapis.com/tensorflow-nightly-public/prod/tensorflow/release/lite/tools/nightly/latest/linux_aarch64_benchmark_model_plus_flex)
* [linux_arm](https://storage.googleapis.com/tensorflow-nightly-public/prod/tensorflow/release/lite/tools/nightly/latest/linux_arm_benchmark_model_plus_flex)
* [android_aarch64](https://storage.googleapis.com/tensorflow-nightly-public/prod/tensorflow/release/lite/tools/nightly/latest/android_aarch64_benchmark_model_plus_flex)
* [android_arm](https://storage.googleapis.com/tensorflow-nightly-public/prod/tensorflow/release/lite/tools/nightly/latest/android_arm_benchmark_model_plus_flex)
To benchmark with [TensorFlow Lite Hexagon delegate](https://www.tensorflow.org/lite/android/delegates/hexagon),
we have also pre-built the required `libhexagon_interface.so` files (see [here](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/delegates/hexagon/README.md)
for details about this file). After downloading the file of the corresponding
platform from the links below, please rename the file to `libhexagon_interface.so`.
* [android_aarch64](https://storage.googleapis.com/tensorflow-nightly-public/prod/tensorflow/release/lite/tools/nightly/latest/android_aarch64_libhexagon_interface.so)
* [android_arm](https://storage.googleapis.com/tensorflow-nightly-public/prod/tensorflow/release/lite/tools/nightly/latest/android_arm_libhexagon_interface.so)
You can also build the native benchmark binary from
[source](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/tools/benchmark)
on your computer.
```shell
bazel build -c opt //tensorflow/lite/tools/benchmark:benchmark_model
```
To build with Android NDK toolchain, you need to set up the build environment
first by following this
[guide](../android/lite_build#set_up_build_environment_without_docker), or use
the docker image as described in this
[guide](../android/lite_build#set_up_build_environment_using_docker).
```shell
bazel build -c opt --config=android_arm64 \
//tensorflow/lite/tools/benchmark:benchmark_model
```
Note: It is a valid approach to push and execute binaries directly on an Android
device for benchmarking, but it can result in subtle (but observable)
differences in performance relative to execution within an actual Android app.
In particular, Android's scheduler tailors behavior based on thread and process
priorities, which differ between a foreground Activity/Application and a regular
background binary executed via `adb shell ...`. This tailored behavior is most
evident when enabling multi-threaded CPU execution with TensorFlow Lite.
Therefore, the Android benchmark app is preferred for performance measurement.
#### Run benchmark
To run benchmarks on your computer, execute the binary from the shell.
```shell
path/to/downloaded_or_built/benchmark_model \
--graph=your_model.tflite \
--num_threads=4
```
You can use the same set of
[parameters](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/tools/benchmark#parameters)
as mentioned above with the native command-line binary.
#### Profiling model ops
The benchmark model binary also allows you to profile model ops and get the
execution times of each operator. To do this, pass the flag
`--enable_op_profiling=true` to `benchmark_model` during invocation. Details are
explained
[here](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/tools/benchmark#profiling-model-operators).
### Native benchmark binary for multiple performance options in a single run
A convenient and simple C++ binary is also provided to
[benchmark multiple performance options](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/tools/benchmark#benchmark-multiple-performance-options-in-a-single-run)
in a single run. This binary is built based on the aforementioned benchmark tool
that could only benchmark a single performance option at a time. They share the
same build/install/run process, but the BUILD target name of this binary is
`benchmark_model_performance_options` and it takes some additional parameters.
An important parameter for this binary is:
`perf_options_list`: `string` (default='all') \
A comma-separated list of TFLite performance options to benchmark.
You can get nightly pre-built binaries for this tool as listed below:
* [linux_x86-64](https://storage.googleapis.com/tensorflow-nightly-public/prod/tensorflow/release/lite/tools/nightly/latest/linux_x86-64_benchmark_model_performance_options)
* [linux_aarch64](https://storage.googleapis.com/tensorflow-nightly-public/prod/tensorflow/release/lite/tools/nightly/latest/linux_aarch64_benchmark_model_performance_options)
* [linux_arm](https://storage.googleapis.com/tensorflow-nightly-public/prod/tensorflow/release/lite/tools/nightly/latest/linux_arm_benchmark_model_performance_options)
* [android_aarch64](https://storage.googleapis.com/tensorflow-nightly-public/prod/tensorflow/release/lite/tools/nightly/latest/android_aarch64_benchmark_model_performance_options)
* [android_arm](https://storage.googleapis.com/tensorflow-nightly-public/prod/tensorflow/release/lite/tools/nightly/latest/android_arm_benchmark_model_performance_options)
### iOS benchmark app
To run benchmarks on iOS device, you need to build the app from
[source](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/tools/benchmark/ios).
Put the TensorFlow Lite model file in the
[benchmark_data](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/tools/benchmark/ios/TFLiteBenchmark/TFLiteBenchmark/benchmark_data)
directory of the source tree and modify the `benchmark_params.json` file. Those
files are packaged into the app and the app reads data from the directory. Visit
the
[iOS benchmark app](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/tools/benchmark/ios)
for detailed instructions.
## Performance benchmarks for well known models
This section lists TensorFlow Lite performance benchmarks when running well
known models on some Android and iOS devices.
### Android performance benchmarks
These performance benchmark numbers were generated with the
[native benchmark binary](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/tools/benchmark).
For Android benchmarks, the CPU affinity is set to use big cores on the device
to reduce variance (see
[details](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/tools/benchmark#reducing-variance-between-runs-on-android)).
It assumes that models were downloaded and unzipped to the
`/data/local/tmp/tflite_models` directory. The benchmark binary is built using
[these instructions](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/tools/benchmark#on-android)
and assumed to be in the `/data/local/tmp` directory.
To run the benchmark:
```sh
adb shell /data/local/tmp/benchmark_model \
--num_threads=4 \
--graph=/data/local/tmp/tflite_models/${GRAPH} \
--warmup_runs=1 \
--num_runs=50
```
To run with nnapi delegate, set `--use_nnapi=true`. To run with GPU delegate,
set `--use_gpu=true`.
The performance values below are measured on Android 10.
<table>
<thead>
<tr>
<th>Model Name</th>
<th>Device </th>
<th>CPU, 4 threads</th>
<th>GPU</th>
<th>NNAPI</th>
</tr>
</thead>
<tr>
<td rowspan = 2>
<a href="https://storage.googleapis.com/download.tensorflow.org/models/mobilenet_v1_2018_08_02/mobilenet_v1_1.0_224.tgz">Mobilenet_1.0_224(float)</a>
</td>
<td>Pixel 3 </td>
<td>23.9 ms</td>
<td>6.45 ms</td>
<td>13.8 ms</td>
</tr>
<tr>
<td>Pixel 4 </td>
<td>14.0 ms</td>
<td>9.0 ms</td>
<td>14.8 ms</td>
</tr>
<tr>
<td rowspan = 2>
<a href="https://storage.googleapis.com/download.tensorflow.org/models/mobilenet_v1_2018_08_02/mobilenet_v1_1.0_224_quant.tgz">Mobilenet_1.0_224 (quant)</a>
</td>
<td>Pixel 3 </td>
<td>13.4 ms</td>
<td>--- </td>
<td>6.0 ms</td>
</tr>
<tr>
<td>Pixel 4 </td>
<td>5.0 ms</td>
<td>--- </td>
<td>3.2 ms</td>
</tr>
<tr>
<td rowspan = 2>
<a href="https://storage.googleapis.com/download.tensorflow.org/models/tflite/model_zoo/upload_20180427/nasnet_mobile_2018_04_27.tgz">NASNet mobile</a>
</td>
<td>Pixel 3 </td>
<td>56 ms</td>
<td>--- </td>
<td>102 ms</td>
</tr>
<tr>
<td>Pixel 4 </td>
<td>34.5 ms</td>
<td>--- </td>
<td>99.0 ms</td>
</tr>
<tr>
<td rowspan = 2>
<a href="https://storage.googleapis.com/download.tensorflow.org/models/tflite/model_zoo/upload_20180427/squeezenet_2018_04_27.tgz">SqueezeNet</a>
</td>
<td>Pixel 3 </td>
<td>35.8 ms</td>
<td>9.5 ms </td>
<td>18.5 ms</td>
</tr>
<tr>
<td>Pixel 4 </td>
<td>23.9 ms</td>
<td>11.1 ms</td>
<td>19.0 ms</td>
</tr>
<tr>
<td rowspan = 2>
<a href="https://storage.googleapis.com/download.tensorflow.org/models/tflite/model_zoo/upload_20180427/inception_resnet_v2_2018_04_27.tgz">Inception_ResNet_V2</a>
</td>
<td>Pixel 3 </td>
<td>422 ms</td>
<td>99.8 ms </td>
<td>201 ms</td>
</tr>
<tr>
<td>Pixel 4 </td>
<td>272.6 ms</td>
<td>87.2 ms</td>
<td>171.1 ms</td>
</tr>
<tr>
<td rowspan = 2>
<a href="https://storage.googleapis.com/download.tensorflow.org/models/tflite/model_zoo/upload_20180427/inception_v4_2018_04_27.tgz">Inception_V4</a>
</td>
<td>Pixel 3 </td>
<td>486 ms</td>
<td>93 ms </td>
<td>292 ms</td>
</tr>
<tr>
<td>Pixel 4 </td>
<td>324.1 ms</td>
<td>97.6 ms</td>
<td>186.9 ms</td>
</tr>
</table>
### iOS performance benchmarks
These performance benchmark numbers were generated with the
[iOS benchmark app](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/tools/benchmark/ios).
To run iOS benchmarks, the benchmark app was modified to include the appropriate
model and `benchmark_params.json` was modified to set `num_threads` to 2. To use
the GPU delegate, `"use_gpu" : "1"` and `"gpu_wait_type" : "aggressive"` options
were also added to `benchmark_params.json`.
<table>
<thead>
<tr>
<th>Model Name</th>
<th>Device </th>
<th>CPU, 2 threads</th>
<th>GPU</th>
</tr>
</thead>
<tr>
<td>
<a href="https://storage.googleapis.com/download.tensorflow.org/models/mobilenet_v1_2018_08_02/mobilenet_v1_1.0_224.tgz">Mobilenet_1.0_224(float)</a>
</td>
<td>iPhone XS </td>
<td>14.8 ms</td>
<td>3.4 ms</td>
</tr>
<tr>
<td>
<a href="https://storage.googleapis.com/download.tensorflow.org/models/mobilenet_v1_2018_08_02/mobilenet_v1_1.0_224_quant.tgz)">Mobilenet_1.0_224 (quant)</a>
</td>
<td>iPhone XS </td>
<td>11 ms</td>
<td>---</td>
</tr>
<tr>
<td>
<a href="https://storage.googleapis.com/download.tensorflow.org/models/tflite/model_zoo/upload_20180427/nasnet_mobile_2018_04_27.tgz">NASNet mobile</a>
</td>
<td>iPhone XS </td>
<td>30.4 ms</td>
<td>---</td>
</tr>
<tr>
<td>
<a href="https://storage.googleapis.com/download.tensorflow.org/models/tflite/model_zoo/upload_20180427/squeezenet_2018_04_27.tgz">SqueezeNet</a>
</td>
<td>iPhone XS </td>
<td>21.1 ms</td>
<td>15.5 ms</td>
</tr>
<tr>
<td>
<a href="https://storage.googleapis.com/download.tensorflow.org/models/tflite/model_zoo/upload_20180427/inception_resnet_v2_2018_04_27.tgz">Inception_ResNet_V2</a>
</td>
<td>iPhone XS </td>
<td>261.1 ms</td>
<td>45.7 ms</td>
</tr>
<tr>
<td>
<a href="https://storage.googleapis.com/download.tensorflow.org/models/tflite/model_zoo/upload_20180427/inception_v4_2018_04_27.tgz">Inception_V4</a>
</td>
<td>iPhone XS </td>
<td>309 ms</td>
<td>54.4 ms</td>
</tr>
</table>
## Trace TensorFlow Lite internals
### Trace TensorFlow Lite internals in Android
Note: This feature is available from Tensorflow Lite v2.4.
Internal events from the TensorFlow Lite interpreter of an Android app can be
captured by
[Android tracing tools](https://developer.android.com/topic/performance/tracing).
They are the same events with Android
[Trace](https://developer.android.com/reference/android/os/Trace) API, so the
captured events from Java/Kotlin code are seen together with TensorFlow Lite
internal events.
Some examples of events are:
* Operator invocation
* Graph modification by delegate
* Tensor allocation
Among different options for capturing traces, this guide covers the Android
Studio CPU Profiler and the System Tracing app. Refer to
[Perfetto command-line tool](https://developer.android.com/studio/command-line/perfetto)
or
[Systrace command-line tool](https://developer.android.com/topic/performance/tracing/command-line)
for other options.
#### Adding trace events in Java code
This is a code snippet from the
[Image Classification](https://github.com/tensorflow/examples/tree/master/lite/examples/image_classification/android)
example app. TensorFlow Lite interpreter runs in the
`recognizeImage/runInference` section. This step is optional but it is useful to
help notice where the inference call is made.
```java
Trace.beginSection("recognizeImage");
...
// Runs the inference call.
Trace.beginSection("runInference");
tflite.run(inputImageBuffer.getBuffer(), outputProbabilityBuffer.getBuffer().rewind());
Trace.endSection();
...
Trace.endSection();
```
#### Enable TensorFlow Lite tracing
To enable TensorFlow Lite tracing, set the Android system property
`debug.tflite.trace` to 1 before starting the Android app.
```shell
adb shell setprop debug.tflite.trace 1
```
If this property has been set when TensorFlow Lite interpreter is initialized,
key events (e.g., operator invocation) from the interpreter will be traced.
After you captured all the traces, disable tracing by setting the property value
to 0.
```shell
adb shell setprop debug.tflite.trace 0
```
#### Android Studio CPU Profiler
Capture traces with the
[Android Studio CPU Profiler](https://developer.android.com/studio/profile/cpu-profiler)
by following the steps below:
1. Select **Run > Profile 'app'** from the top menus.
2. Click anywhere in CPU timeline when the Profiler window appears.
3. Select 'Trace System Calls' among CPU Profiling modes.
![Select 'Trace System Calls'](images/as_select_profiling_mode.png)
4. Press 'Record' button.
5. Press 'Stop' button.
6. Investigate the trace result.
![Android Studio trace](images/as_traces.png)
In this example, you can see the hierarchy of events in a thread and statistics
for each operator time and also see the data flow of the whole app among
threads.
#### System Tracing app
Capture traces without Android Studio by following the steps detailed in
[System Tracing app](https://developer.android.com/topic/performance/tracing/on-device).
In this example, the same TFLite events were captured and saved to the Perfetto
or Systrace format depending on the version of Android device. The captured
trace files can be opened in the [Perfetto UI](https://ui.perfetto.dev/#!/).
![Perfetto trace](images/perfetto_traces.png)
### Trace TensorFlow Lite internals in iOS
Note: This feature is available from Tensorflow Lite v2.5.
Internal events from the TensorFlow Lite interpreter of an iOS app can be
captured by
[Instruments](https://developer.apple.com/library/archive/documentation/ToolsLanguages/Conceptual/Xcode_Overview/MeasuringPerformance.html#//apple_ref/doc/uid/TP40010215-CH60-SW1)
tool included with Xcode. They are the iOS
[signpost](https://developer.apple.com/documentation/os/logging/recording_performance_data)
events, so the captured events from Swift/Objective-C code are seen together
with TensorFlow Lite internal events.
Some examples of events are:
* Operator invocation
* Graph modification by delegate
* Tensor allocation
#### Enable TensorFlow Lite tracing
Set the environment variable `debug.tflite.trace` by following the steps below:
1. Select **Product > Scheme > Edit Scheme...** from the top menus of Xcode.
2. Click 'Profile' in the left pane.
3. Deselect 'Use the Run action's arguments and environment variables'
checkbox.
4. Add `debug.tflite.trace` under 'Environment Variables' section.
![Set environment variable](images/xcode_profile_environment.png)
If you want to exclude TensorFlow Lite events when profiling the iOS app,
disable tracing by removing the environment variable.
#### XCode Instruments
Capture traces by following the steps below:
1. Select **Product > Profile** from the top menus of Xcode.
2. Click **Logging** among profiling templates when Instruments tool launches.
3. Press 'Start' button.
4. Press 'Stop' button.
5. Click 'os_signpost' to expand OS Logging subsystem items.
6. Click 'org.tensorflow.lite' OS Logging subsystem.
7. Investigate the trace result.
![Xcode Instruments trace](images/xcode_traces.png)
In this example, you can see the hierarchy of events and statistics for each
operator time.
### Using the tracing data
The tracing data allows you to identify performance bottlenecks.
Here are some examples of insights that you can get from the profiler and
potential solutions to improve performance:
* If the number of available CPU cores is smaller than the number of inference
threads, then the CPU scheduling overhead can lead to subpar performance.
You can reschedule other CPU intensive tasks in your application to avoid
overlapping with your model inference or tweak the number of interpreter
threads.
* If the operators are not fully delegated, then some parts of the model graph
are executed on the CPU rather than the expected hardware accelerator. You
can substitute the unsupported operators with similar supported operators.
tensorflow/lite/g3doc/performance/model_optimization.md
+211
View File
@@ -0,0 +1,211 @@
# Model optimization
Edge devices often have limited memory or computational power. Various
optimizations can be applied to models so that they can be run within these
constraints. In addition, some optimizations allow the use of specialized
hardware for accelerated inference.
TensorFlow Lite and the
[TensorFlow Model Optimization Toolkit](https://www.tensorflow.org/model_optimization)
provide tools to minimize the complexity of optimizing inference.
It's recommended that you consider model optimization during your application
development process. This document outlines some best practices for optimizing
TensorFlow models for deployment to edge hardware.
## Why models should be optimized
There are several main ways model optimization can help with application
development.
### Size reduction
Some forms of optimization can be used to reduce the size of a model. Smaller
models have the following benefits:
- **Smaller storage size:** Smaller models occupy less storage space on your
users' devices. For example, an Android app using a smaller model will take
up less storage space on a user's mobile device.
- **Smaller download size:** Smaller models require less time and bandwidth to
download to users' devices.
- **Less memory usage:** Smaller models use less RAM when they are run, which
frees up memory for other parts of your application to use, and can
translate to better performance and stability.
Quantization can reduce the size of a model in all of these cases, potentially
at the expense of some accuracy. Pruning and clustering can reduce the size of a
model for download by making it more easily compressible.
### Latency reduction
*Latency* is the amount of time it takes to run a single inference with a given
model. Some forms of optimization can reduce the amount of computation required
to run inference using a model, resulting in lower latency. Latency can also
have an impact on power consumption.
Currently, quantization can be used to reduce latency by simplifying the
calculations that occur during inference, potentially at the expense of some
accuracy.
### Accelerator compatibility
Some hardware accelerators, such as the
[Edge TPU](https://cloud.google.com/edge-tpu/), can run inference extremely fast
with models that have been correctly optimized.
Generally, these types of devices require models to be quantized in a specific
way. See each hardware accelerator's documentation to learn more about their
requirements.
## Trade-offs
Optimizations can potentially result in changes in model accuracy, which must be
considered during the application development process.
The accuracy changes depend on the individual model being optimized, and are
difficult to predict ahead of time. Generally, models that are optimized for
size or latency will lose a small amount of accuracy. Depending on your
application, this may or may not impact your users' experience. In rare cases,
certain models may gain some accuracy as a result of the optimization process.
## Types of optimization
TensorFlow Lite currently supports optimization via quantization, pruning and
clustering.
These are part of the
[TensorFlow Model Optimization Toolkit](https://www.tensorflow.org/model_optimization),
which provides resources for model optimization techniques that are compatible
with TensorFlow Lite.
### Quantization
[Quantization](https://www.tensorflow.org/model_optimization/guide/quantization/post_training)
works by reducing the precision of the numbers used to represent a model's
parameters, which by default are 32-bit floating point numbers. This results in
a smaller model size and faster computation.
The following types of quantization are available in TensorFlow Lite:
Technique | Data requirements | Size reduction | Accuracy | Supported hardware
------------------------------------------------------------------------------------------------------- | -------------------------------- | -------------- | --------------------------- | ------------------
[Post-training float16 quantization](post_training_float16_quant.ipynb) | No data | Up to 50% | Insignificant accuracy loss | CPU, GPU
[Post-training dynamic range quantization](post_training_quant.ipynb) | No data | Up to 75% | Smallest accuracy loss | CPU, GPU (Android)
[Post-training integer quantization](post_training_integer_quant.ipynb) | Unlabelled representative sample | Up to 75% | Small accuracy loss | CPU, GPU (Android), EdgeTPU, Hexagon DSP
[Quantization-aware training](http://www.tensorflow.org/model_optimization/guide/quantization/training) | Labelled training data | Up to 75% | Smallest accuracy loss | CPU, GPU (Android), EdgeTPU, Hexagon DSP
The following decision tree helps you select the quantization schemes you might
want to use for your model, simply based on the expected model size and
accuracy.
![quantization-decision-tree](images/quantization_decision_tree.png)
Below are the latency and accuracy results for post-training quantization and
quantization-aware training on a few models. All latency numbers are measured on
Pixel 2 devices using a single big core CPU. As the toolkit improves, so will
the numbers here:
<figure>
<table>
<tr>
<th>Model</th>
<th>Top-1 Accuracy (Original) </th>
<th>Top-1 Accuracy (Post Training Quantized) </th>
<th>Top-1 Accuracy (Quantization Aware Training) </th>
<th>Latency (Original) (ms) </th>
<th>Latency (Post Training Quantized) (ms) </th>
<th>Latency (Quantization Aware Training) (ms) </th>
<th> Size (Original) (MB)</th>
<th> Size (Optimized) (MB)</th>
</tr> <tr><td>Mobilenet-v1-1-224</td><td>0.709</td><td>0.657</td><td>0.70</td>
<td>124</td><td>112</td><td>64</td><td>16.9</td><td>4.3</td></tr>
<tr><td>Mobilenet-v2-1-224</td><td>0.719</td><td>0.637</td><td>0.709</td>
<td>89</td><td>98</td><td>54</td><td>14</td><td>3.6</td></tr>
<tr><td>Inception_v3</td><td>0.78</td><td>0.772</td><td>0.775</td>
<td>1130</td><td>845</td><td>543</td><td>95.7</td><td>23.9</td></tr>
<tr><td>Resnet_v2_101</td><td>0.770</td><td>0.768</td><td>N/A</td>
<td>3973</td><td>2868</td><td>N/A</td><td>178.3</td><td>44.9</td></tr>
</table>
<figcaption>
<b>Table 1</b> Benefits of model quantization for select CNN models
</figcaption>
</figure>
### Full integer quantization with int16 activations and int8 weights
[Quantization with int16 activations](https://www.tensorflow.org/model_optimization/guide/quantization/post_training)
is a full integer quantization scheme with activations in int16 and weights in
int8. This mode can improve accuracy of the quantized model in comparison to the
full integer quantization scheme with both activations and weights in int8
keeping a similar model size. It is recommended when activations are sensitive
to the quantization.
<i>NOTE:</i> Currently only non-optimized reference kernel implementations are
available in TFLite for this quantization scheme, so by default the performance
will be slow compared to int8 kernels. Full advantages of this mode can
currently be accessed via specialised hardware, or custom software.
Below are the accuracy results for some models that benefit from this mode.
<figure>
<table>
<tr>
<th>Model</th>
<th>Accuracy metric type </th>
<th>Accuracy (float32 activations) </th>
<th>Accuracy (int8 activations) </th>
<th>Accuracy (int16 activations) </th>
</tr> <tr><td>Wav2letter</td><td>WER</td><td>6.7%</td><td>7.7%</td>
<td>7.2%</td></tr>
<tr><td>DeepSpeech 0.5.1 (unrolled)</td><td>CER</td><td>6.13%</td><td>43.67%</td>
<td>6.52%</td></tr>
<tr><td>YoloV3</td><td>mAP(IOU=0.5)</td><td>0.577</td><td>0.563</td>
<td>0.574</td></tr>
<tr><td>MobileNetV1</td><td>Top-1 Accuracy</td><td>0.7062</td><td>0.694</td>
<td>0.6936</td></tr>
<tr><td>MobileNetV2</td><td>Top-1 Accuracy</td><td>0.718</td><td>0.7126</td>
<td>0.7137</td></tr>
<tr><td>MobileBert</td><td>F1(Exact match)</td><td>88.81(81.23)</td><td>2.08(0)</td>
<td>88.73(81.15)</td></tr>
</table>
<figcaption>
<b>Table 2</b> Benefits of model quantization with int16 activations
</figcaption>
</figure>
### Pruning
[Pruning](https://www.tensorflow.org/model_optimization/guide/pruning) works by
removing parameters within a model that have only a minor impact on its
predictions. Pruned models are the same size on disk, and have the same runtime
latency, but can be compressed more effectively. This makes pruning a useful
technique for reducing model download size.
In the future, TensorFlow Lite will provide latency reduction for pruned models.
### Clustering
[Clustering](https://www.tensorflow.org/model_optimization/guide/clustering)
works by grouping the weights of each layer in a model into a predefined number
of clusters, then sharing the centroid values for the weights belonging to each
individual cluster. This reduces the number of unique weight values in a model,
thus reducing its complexity.
As a result, clustered models can be compressed more effectively, providing
deployment benefits similar to pruning.
## Development workflow
As a starting point, check if the models in
[hosted models](../guide/hosted_models.md) can work for your application. If
not, we recommend that users start with the
[post-training quantization tool](post_training_quantization.md) since this is
broadly applicable and does not require training data.
For cases where the accuracy and latency targets are not met, or hardware
accelerator support is important,
[quantization-aware training](https://www.tensorflow.org/model_optimization/guide/quantization/training){:.external}
is the better option. See additional optimization techniques under the
[TensorFlow Model Optimization Toolkit](https://www.tensorflow.org/model_optimization).
If you want to further reduce your model size, you can try [pruning](#pruning)
and/or [clustering](#clustering) prior to quantizing your models.
tensorflow/lite/g3doc/performance/post_training_float16_quant.ipynb
+544
View File
@@ -0,0 +1,544 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "c8Cx-rUMVX25"
},
"source": [
"##### Copyright 2024 The AI Edge Authors."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "I9sUhVL_VZNO"
},
"outputs": [],
"source": [
"#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6Y8E0lw5eYWm"
},
"source": [
"# Post-training float16 quantization"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "CGuqeuPSVNo-"
},
"source": [
"<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
" <td>\n",
" <a target=\"_blank\" href=\"https://www.tensorflow.org/lite/performance/post_training_float16_quant\"><img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" />View on TensorFlow.org</a>\n",
" </td>\n",
" <td>\n",
" <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/performance/post_training_float16_quant.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
" </td>\n",
" <td>\n",
" <a target=\"_blank\" href=\"https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/performance/post_training_float16_quant.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://storage.googleapis.com/tensorflow_docs/tensorflow/tensorflow/lite/g3doc/performance/post_training_float16_quant.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\" />Download notebook</a>\n",
" </td>\n",
"</table>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "BTC1rDAuei_1"
},
"source": [
"## Overview\n",
"\n",
"[TensorFlow Lite](https://www.tensorflow.org/lite/) now supports\n",
"converting weights to 16-bit floating point values during model conversion from TensorFlow to TensorFlow Lite's flat buffer format. This results in a 2x reduction in model size. Some hardware, like GPUs, can compute natively in this reduced precision arithmetic, realizing a speedup over traditional floating point execution. The Tensorflow Lite GPU delegate can be configured to run in this way. However, a model converted to float16 weights can still run on the CPU without additional modification: the float16 weights are upsampled to float32 prior to the first inference. This permits a significant reduction in model size in exchange for a minimal impacts to latency and accuracy.\n",
"\n",
"In this tutorial, you train an MNIST model from scratch, check its accuracy in TensorFlow, and then convert the model into a Tensorflow Lite flatbuffer\n",
"with float16 quantization. Finally, check the accuracy of the converted model and compare it to the original float32 model."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2XsEP17Zelz9"
},
"source": [
"## Build an MNIST model"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dDqqUIZjZjac"
},
"source": [
"### Setup"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "gyqAw1M9lyab"
},
"outputs": [],
"source": [
"import logging\n",
"logging.getLogger(\"tensorflow\").setLevel(logging.DEBUG)\n",
"\n",
"import tensorflow as tf\n",
"from tensorflow import keras\n",
"import numpy as np\n",
"import pathlib"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "eQ6Q0qqKZogR"
},
"source": [
"### Train and export the model"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "hWSAjQWagIHl"
},
"outputs": [],
"source": [
"# Load MNIST dataset\n",
"mnist = keras.datasets.mnist\n",
"(train_images, train_labels), (test_images, test_labels) = mnist.load_data()\n",
"\n",
"# Normalize the input image so that each pixel value is between 0 to 1.\n",
"train_images = train_images / 255.0\n",
"test_images = test_images / 255.0\n",
"\n",
"# Define the model architecture\n",
"model = keras.Sequential([\n",
" keras.layers.InputLayer(input_shape=(28, 28)),\n",
" keras.layers.Reshape(target_shape=(28, 28, 1)),\n",
" keras.layers.Conv2D(filters=12, kernel_size=(3, 3), activation=tf.nn.relu),\n",
" keras.layers.MaxPooling2D(pool_size=(2, 2)),\n",
" keras.layers.Flatten(),\n",
" keras.layers.Dense(10)\n",
"])\n",
"\n",
"# Train the digit classification model\n",
"model.compile(optimizer='adam',\n",
" loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
" metrics=['accuracy'])\n",
"model.fit(\n",
" train_images,\n",
" train_labels,\n",
" epochs=1,\n",
" validation_data=(test_images, test_labels)\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5NMaNZQCkW9X"
},
"source": [
"For the example, you trained the model for just a single epoch, so it only trains to ~96% accuracy."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "xl8_fzVAZwOh"
},
"source": [
"### Convert to a TensorFlow Lite model\n",
"\n",
"Using the TensorFlow Lite [Converter](https://www.tensorflow.org/lite/models/convert), you can now convert the trained model into a TensorFlow Lite model.\n",
"\n",
"Now load the model using the `TFLiteConverter`:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "_i8B2nDZmAgQ"
},
"outputs": [],
"source": [
"converter = tf.lite.TFLiteConverter.from_keras_model(model)\n",
"tflite_model = converter.convert()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "F2o2ZfF0aiCx"
},
"source": [
"Write it out to a `.tflite` file:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "vptWZq2xnclo"
},
"outputs": [],
"source": [
"tflite_models_dir = pathlib.Path(\"/tmp/mnist_tflite_models/\")\n",
"tflite_models_dir.mkdir(exist_ok=True, parents=True)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Ie9pQaQrn5ue"
},
"outputs": [],
"source": [
"tflite_model_file = tflite_models_dir/\"mnist_model.tflite\"\n",
"tflite_model_file.write_bytes(tflite_model)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7BONhYtYocQY"
},
"source": [
"To instead quantize the model to float16 on export, first set the `optimizations` flag to use default optimizations. Then specify that float16 is the supported type on the target platform:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "HEZ6ET1AHAS3"
},
"outputs": [],
"source": [
"converter.optimizations = [tf.lite.Optimize.DEFAULT]\n",
"converter.target_spec.supported_types = [tf.float16]"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "xW84iMYjHd9t"
},
"source": [
"Finally, convert the model like usual. Note, by default the converted model will still use float input and outputs for invocation convenience."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "yuNfl3CoHNK3"
},
"outputs": [],
"source": [
"tflite_fp16_model = converter.convert()\n",
"tflite_model_fp16_file = tflite_models_dir/\"mnist_model_quant_f16.tflite\"\n",
"tflite_model_fp16_file.write_bytes(tflite_fp16_model)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "PhMmUTl4sbkz"
},
"source": [
"Note how the resulting file is approximately `1/2` the size."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "JExfcfLDscu4"
},
"outputs": [],
"source": [
"!ls -lh {tflite_models_dir}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "L8lQHMp_asCq"
},
"source": [
"## Run the TensorFlow Lite models"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "-5l6-ciItvX6"
},
"source": [
"Run the TensorFlow Lite model using the Python TensorFlow Lite Interpreter."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Ap_jE7QRvhPf"
},
"source": [
"### Load the model into the interpreters"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Jn16Rc23zTss"
},
"outputs": [],
"source": [
"interpreter = tf.lite.Interpreter(model_path=str(tflite_model_file))\n",
"interpreter.allocate_tensors()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "J8Pztk1mvNVL"
},
"outputs": [],
"source": [
"interpreter_fp16 = tf.lite.Interpreter(model_path=str(tflite_model_fp16_file))\n",
"interpreter_fp16.allocate_tensors()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2opUt_JTdyEu"
},
"source": [
"### Test the models on one image"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "AKslvo2kwWac"
},
"outputs": [],
"source": [
"test_image = np.expand_dims(test_images[0], axis=0).astype(np.float32)\n",
"\n",
"input_index = interpreter.get_input_details()[0][\"index\"]\n",
"output_index = interpreter.get_output_details()[0][\"index\"]\n",
"\n",
"interpreter.set_tensor(input_index, test_image)\n",
"interpreter.invoke()\n",
"predictions = interpreter.get_tensor(output_index)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "XZClM2vo3_bm"
},
"outputs": [],
"source": [
"import matplotlib.pylab as plt\n",
"\n",
"plt.imshow(test_images[0])\n",
"template = \"True:{true}, predicted:{predict}\"\n",
"_ = plt.title(template.format(true= str(test_labels[0]),\n",
" predict=str(np.argmax(predictions[0]))))\n",
"plt.grid(False)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "3gwhv4lKbYZ4"
},
"outputs": [],
"source": [
"test_image = np.expand_dims(test_images[0], axis=0).astype(np.float32)\n",
"\n",
"input_index = interpreter_fp16.get_input_details()[0][\"index\"]\n",
"output_index = interpreter_fp16.get_output_details()[0][\"index\"]\n",
"\n",
"interpreter_fp16.set_tensor(input_index, test_image)\n",
"interpreter_fp16.invoke()\n",
"predictions = interpreter_fp16.get_tensor(output_index)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "CIH7G_MwbY2x"
},
"outputs": [],
"source": [
"plt.imshow(test_images[0])\n",
"template = \"True:{true}, predicted:{predict}\"\n",
"_ = plt.title(template.format(true= str(test_labels[0]),\n",
" predict=str(np.argmax(predictions[0]))))\n",
"plt.grid(False)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "LwN7uIdCd8Gw"
},
"source": [
"### Evaluate the models"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "05aeAuWjvjPx"
},
"outputs": [],
"source": [
"# A helper function to evaluate the TF Lite model using \"test\" dataset.\n",
"def evaluate_model(interpreter):\n",
" input_index = interpreter.get_input_details()[0][\"index\"]\n",
" output_index = interpreter.get_output_details()[0][\"index\"]\n",
"\n",
" # Run predictions on every image in the \"test\" dataset.\n",
" prediction_digits = []\n",
" for test_image in test_images:\n",
" # Pre-processing: add batch dimension and convert to float32 to match with\n",
" # the model's input data format.\n",
" test_image = np.expand_dims(test_image, axis=0).astype(np.float32)\n",
" interpreter.set_tensor(input_index, test_image)\n",
"\n",
" # Run inference.\n",
" interpreter.invoke()\n",
"\n",
" # Post-processing: remove batch dimension and find the digit with highest\n",
" # probability.\n",
" output = interpreter.tensor(output_index)\n",
" digit = np.argmax(output()[0])\n",
" prediction_digits.append(digit)\n",
"\n",
" # Compare prediction results with ground truth labels to calculate accuracy.\n",
" accurate_count = 0\n",
" for index in range(len(prediction_digits)):\n",
" if prediction_digits[index] == test_labels[index]:\n",
" accurate_count += 1\n",
" accuracy = accurate_count * 1.0 / len(prediction_digits)\n",
"\n",
" return accuracy"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "T5mWkSbMcU5z"
},
"outputs": [],
"source": [
"print(evaluate_model(interpreter))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Km3cY9ry8ZlG"
},
"source": [
"Repeat the evaluation on the float16 quantized model to obtain:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "-9cnwiPp6EGm"
},
"outputs": [],
"source": [
"# NOTE: Colab runs on server CPUs. At the time of writing this, TensorFlow Lite\n",
"# doesn't have super optimized server CPU kernels. For this reason this may be\n",
"# slower than the above float interpreter. But for mobile CPUs, considerable\n",
"# speedup can be observed.\n",
"print(evaluate_model(interpreter_fp16))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "L7lfxkor8pgv"
},
"source": [
"In this example, you have quantized a model to float16 with no difference in the accuracy.\n",
"\n",
"It's also possible to evaluate the fp16 quantized model on the GPU. To perform all arithmetic with the reduced precision values, be sure to create the `TfLiteGPUDelegateOptions` struct in your app and set `precision_loss_allowed` to `1`, like this:\n",
"\n",
"```\n",
"//Prepare GPU delegate.\n",
"const TfLiteGpuDelegateOptions options = {\n",
" .metadata = NULL,\n",
" .compile_options = {\n",
" .precision_loss_allowed = 1, // FP16\n",
" .preferred_gl_object_type = TFLITE_GL_OBJECT_TYPE_FASTEST,\n",
" .dynamic_batch_enabled = 0, // Not fully functional yet\n",
" },\n",
"};\n",
"```\n",
"\n",
"Detailed documentation on the TFLite GPU delegate and how to use it in your application can be found [here](https://www.tensorflow.org/lite/performance/gpu_advanced?source=post_page---------------------------)"
]
}
],
"metadata": {
"colab": {
"collapsed_sections": [],
"name": "post_training_float16_quant.ipynb",
"provenance": [],
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
tensorflow/lite/g3doc/performance/post_training_integer_quant.ipynb
+703
View File
@@ -0,0 +1,703 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "_DDaAex5Q7u-"
},
"source": [
"##### Copyright 2024 The AI Edge Authors."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "W1dWWdNHQ9L0"
},
"outputs": [],
"source": [
"#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6Y8E0lw5eYWm"
},
"source": [
"# Post-training integer quantization"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "CIGrZZPTZVeO"
},
"source": [
"<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
" <td>\n",
" <a target=\"_blank\" href=\"https://www.tensorflow.org/lite/performance/post_training_integer_quant\"><img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" />View on TensorFlow.org</a>\n",
" </td>\n",
" <td>\n",
" <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/performance/post_training_integer_quant.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
" </td>\n",
" <td>\n",
" <a target=\"_blank\" href=\"https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/performance/post_training_integer_quant.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://storage.googleapis.com/tensorflow_docs/tensorflow/tensorflow/lite/g3doc/performance/post_training_integer_quant.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\" />Download notebook</a>\n",
" </td>\n",
"</table>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "BTC1rDAuei_1"
},
"source": [
"## Overview\n",
"\n",
"Integer quantization is an optimization strategy that converts 32-bit floating-point numbers (such as weights and activation outputs) to the nearest 8-bit fixed-point numbers. This results in a smaller model and increased inferencing speed, which is valuable for low-power devices such as [microcontrollers](https://www.tensorflow.org/lite/microcontrollers). This data format is also required by integer-only accelerators such as the [Edge TPU](https://coral.ai/).\n",
"\n",
"In this tutorial, you'll train an MNIST model from scratch, convert it into a Tensorflow Lite file, and quantize it using [post-training quantization](https://www.tensorflow.org/lite/performance/post_training_quantization). Finally, you'll check the accuracy of the converted model and compare it to the original float model.\n",
"\n",
"You actually have several options as to how much you want to quantize a model. In this tutorial, you'll perform \"full integer quantization,\" which converts all weights and activation outputs into 8-bit integer data—whereas other strategies may leave some amount of data in floating-point.\n",
"\n",
"To learn more about the various quantization strategies, read about [TensorFlow Lite model optimization](https://www.tensorflow.org/lite/performance/model_optimization).\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dDqqUIZjZjac"
},
"source": [
"## Setup"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "I0nR5AMEWq0H"
},
"source": [
"In order to quantize both the input and output tensors, we need to use APIs added in TensorFlow 2.3:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "WsN6s5L1ieNl"
},
"outputs": [],
"source": [
"import logging\n",
"logging.getLogger(\"tensorflow\").setLevel(logging.DEBUG)\n",
"\n",
"import tensorflow as tf\n",
"import numpy as np\n",
"print(\"TensorFlow version: \", tf.__version__)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2XsEP17Zelz9"
},
"source": [
"## Generate a TensorFlow Model"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5NMaNZQCkW9X"
},
"source": [
"We'll build a simple model to classify numbers from the [MNIST dataset](https://www.tensorflow.org/datasets/catalog/mnist).\n",
"\n",
"This training won't take long because you're training the model for just a 5 epochs, which trains to about ~98% accuracy."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "eMsw_6HujaqM"
},
"outputs": [],
"source": [
"# Load MNIST dataset\n",
"mnist = tf.keras.datasets.mnist\n",
"(train_images, train_labels), (test_images, test_labels) = mnist.load_data()\n",
"\n",
"# Normalize the input image so that each pixel value is between 0 to 1.\n",
"train_images = train_images.astype(np.float32) / 255.0\n",
"test_images = test_images.astype(np.float32) / 255.0\n",
"\n",
"# Define the model architecture\n",
"model = tf.keras.Sequential([\n",
" tf.keras.layers.InputLayer(input_shape=(28, 28)),\n",
" tf.keras.layers.Reshape(target_shape=(28, 28, 1)),\n",
" tf.keras.layers.Conv2D(filters=12, kernel_size=(3, 3), activation='relu'),\n",
" tf.keras.layers.MaxPooling2D(pool_size=(2, 2)),\n",
" tf.keras.layers.Flatten(),\n",
" tf.keras.layers.Dense(10)\n",
"])\n",
"\n",
"# Train the digit classification model\n",
"model.compile(optimizer='adam',\n",
" loss=tf.keras.losses.SparseCategoricalCrossentropy(\n",
" from_logits=True),\n",
" metrics=['accuracy'])\n",
"model.fit(\n",
" train_images,\n",
" train_labels,\n",
" epochs=5,\n",
" validation_data=(test_images, test_labels)\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "KuTEoGFYd8aM"
},
"source": [
"## Convert to a TensorFlow Lite model"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "xl8_fzVAZwOh"
},
"source": [
"Now you can convert the trained model to TensorFlow Lite format using the TensorFlow Lite [Converter](https://www.tensorflow.org/lite/models/convert), and apply varying degrees of quantization.\n",
"\n",
"Beware that some versions of quantization leave some of the data in float format. So the following sections show each option with increasing amounts of quantization, until we get a model that's entirely int8 or uint8 data. (Notice we duplicate some code in each section so you can see all the quantization steps for each option.)\n",
"\n",
"First, here's a converted model with no quantization:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "_i8B2nDZmAgQ"
},
"outputs": [],
"source": [
"converter = tf.lite.TFLiteConverter.from_keras_model(model)\n",
"\n",
"tflite_model = converter.convert()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7BONhYtYocQY"
},
"source": [
"It's now a TensorFlow Lite model, but it's still using 32-bit float values for all parameter data."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "jPYZwgZTwJMT"
},
"source": [
"### Convert using dynamic range quantization\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Hjvq1vpJd4U_"
},
"source": [
"Now let's enable the default `optimizations` flag to quantize all fixed parameters (such as weights):"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "HEZ6ET1AHAS3"
},
"outputs": [],
"source": [
"converter = tf.lite.TFLiteConverter.from_keras_model(model)\n",
"converter.optimizations = [tf.lite.Optimize.DEFAULT]\n",
"\n",
"tflite_model_quant = converter.convert()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "o5wuE-RcdX_3"
},
"source": [
"The model is now a bit smaller with quantized weights, but other variable data is still in float format."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "UgKDdnHQEhpb"
},
"source": [
"### Convert using float fallback quantization"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "rTe8avZJHMDO"
},
"source": [
"To quantize the variable data (such as model input/output and intermediates between layers), you need to provide a [`RepresentativeDataset`](https://www.tensorflow.org/api_docs/python/tf/lite/RepresentativeDataset). This is a generator function that provides a set of input data that's large enough to represent typical values. It allows the converter to estimate a dynamic range for all the variable data. (The dataset does not need to be unique compared to the training or evaluation dataset.)\n",
"To support multiple inputs, each representative data point is a list and elements in the list are fed to the model according to their indices.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "FiwiWU3gHdkW"
},
"outputs": [],
"source": [
"def representative_data_gen():\n",
" for input_value in tf.data.Dataset.from_tensor_slices(train_images).batch(1).take(100):\n",
" # Model has only one input so each data point has one element.\n",
" yield [input_value]\n",
"\n",
"converter = tf.lite.TFLiteConverter.from_keras_model(model)\n",
"converter.optimizations = [tf.lite.Optimize.DEFAULT]\n",
"converter.representative_dataset = representative_data_gen\n",
"\n",
"tflite_model_quant = converter.convert()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_GC3HFlptf7x"
},
"source": [
"Now all weights and variable data are quantized, and the model is significantly smaller compared to the original TensorFlow Lite model.\n",
"\n",
"However, to maintain compatibility with applications that traditionally use float model input and output tensors, the TensorFlow Lite Converter leaves the model input and output tensors in float:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "id1OEKFELQwp"
},
"outputs": [],
"source": [
"interpreter = tf.lite.Interpreter(model_content=tflite_model_quant)\n",
"input_type = interpreter.get_input_details()[0]['dtype']\n",
"print('input: ', input_type)\n",
"output_type = interpreter.get_output_details()[0]['dtype']\n",
"print('output: ', output_type)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "RACBJuj2XO8x"
},
"source": [
"That's usually good for compatibility, but it won't be compatible with devices that perform only integer-based operations, such as the Edge TPU.\n",
"\n",
"Additionally, the above process may leave an operation in float format if TensorFlow Lite doesn't include a quantized implementation for that operation. This strategy allows conversion to complete so you have a smaller and more efficient model, but again, it won't be compatible with integer-only hardware. (All ops in this MNIST model have a quantized implementation.)\n",
"\n",
"So to ensure an end-to-end integer-only model, you need a couple more parameters..."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "FQgTqbvPvxGJ"
},
"source": [
"### Convert using integer-only quantization"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "mwR9keYAwArA"
},
"source": [
"To quantize the input and output tensors, and make the converter throw an error if it encounters an operation it cannot quantize, convert the model again with some additional parameters:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "kzjEjcDs3BHa"
},
"outputs": [],
"source": [
"def representative_data_gen():\n",
" for input_value in tf.data.Dataset.from_tensor_slices(train_images).batch(1).take(100):\n",
" yield [input_value]\n",
"\n",
"converter = tf.lite.TFLiteConverter.from_keras_model(model)\n",
"converter.optimizations = [tf.lite.Optimize.DEFAULT]\n",
"converter.representative_dataset = representative_data_gen\n",
"# Ensure that if any ops can't be quantized, the converter throws an error\n",
"converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]\n",
"# Set the input and output tensors to uint8 (APIs added in r2.3)\n",
"converter.inference_input_type = tf.uint8\n",
"converter.inference_output_type = tf.uint8\n",
"\n",
"tflite_model_quant = converter.convert()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "wYd6NxD03yjB"
},
"source": [
"The internal quantization remains the same as above, but you can see the input and output tensors are now integer format:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "PaNkOS-twz4k"
},
"outputs": [],
"source": [
"interpreter = tf.lite.Interpreter(model_content=tflite_model_quant)\n",
"input_type = interpreter.get_input_details()[0]['dtype']\n",
"print('input: ', input_type)\n",
"output_type = interpreter.get_output_details()[0]['dtype']\n",
"print('output: ', output_type)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "TO17AP84wzBb"
},
"source": [
"Now you have an integer quantized model that uses integer data for the model's input and output tensors, so it's compatible with integer-only hardware such as the [Edge TPU](https://coral.ai)."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "sse224YJ4KMm"
},
"source": [
"### Save the models as files"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "4_9nZ4nv4b9P"
},
"source": [
"You'll need a `.tflite` file to deploy your model on other devices. So let's save the converted models to files and then load them when we run inferences below."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "BEY59dC14uRv"
},
"outputs": [],
"source": [
"import pathlib\n",
"\n",
"tflite_models_dir = pathlib.Path(\"/tmp/mnist_tflite_models/\")\n",
"tflite_models_dir.mkdir(exist_ok=True, parents=True)\n",
"\n",
"# Save the unquantized/float model:\n",
"tflite_model_file = tflite_models_dir/\"mnist_model.tflite\"\n",
"tflite_model_file.write_bytes(tflite_model)\n",
"# Save the quantized model:\n",
"tflite_model_quant_file = tflite_models_dir/\"mnist_model_quant.tflite\"\n",
"tflite_model_quant_file.write_bytes(tflite_model_quant)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "9t9yaTeF9fyM"
},
"source": [
"## Run the TensorFlow Lite models"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "L8lQHMp_asCq"
},
"source": [
"Now we'll run inferences using the TensorFlow Lite [`Interpreter`](https://www.tensorflow.org/api_docs/python/tf/lite/Interpreter) to compare the model accuracies.\n",
"\n",
"First, we need a function that runs inference with a given model and images, and then returns the predictions:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "X092SbeWfd1A"
},
"outputs": [],
"source": [
"# Helper function to run inference on a TFLite model\n",
"def run_tflite_model(tflite_file, test_image_indices):\n",
" global test_images\n",
"\n",
" # Initialize the interpreter\n",
" interpreter = tf.lite.Interpreter(model_path=str(tflite_file))\n",
" interpreter.allocate_tensors()\n",
"\n",
" input_details = interpreter.get_input_details()[0]\n",
" output_details = interpreter.get_output_details()[0]\n",
"\n",
" predictions = np.zeros((len(test_image_indices),), dtype=int)\n",
" for i, test_image_index in enumerate(test_image_indices):\n",
" test_image = test_images[test_image_index]\n",
"\n",
" # Check if the input type is quantized, then rescale input data to uint8\n",
" if input_details['dtype'] == np.uint8:\n",
" input_scale, input_zero_point = input_details[\"quantization\"]\n",
" test_image = test_image / input_scale + input_zero_point\n",
"\n",
" test_image = np.expand_dims(test_image, axis=0).astype(input_details[\"dtype\"])\n",
" interpreter.set_tensor(input_details[\"index\"], test_image)\n",
" interpreter.invoke()\n",
" output = interpreter.get_tensor(output_details[\"index\"])[0]\n",
"\n",
" predictions[i] = output.argmax()\n",
"\n",
" return predictions\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2opUt_JTdyEu"
},
"source": [
"### Test the models on one image\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "QpPpFPaz7eEM"
},
"source": [
"Now we'll compare the performance of the float model and quantized model:\n",
"+ `tflite_model_file` is the original TensorFlow Lite model with floating-point data.\n",
"+ `tflite_model_quant_file` is the last model we converted using integer-only quantization (it uses uint8 data for input and output).\n",
"\n",
"Let's create another function to print our predictions:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "zR2cHRUcUZ6e"
},
"outputs": [],
"source": [
"import matplotlib.pylab as plt\n",
"\n",
"# Change this to test a different image\n",
"test_image_index = 1\n",
"\n",
"## Helper function to test the models on one image\n",
"def test_model(tflite_file, test_image_index, model_type):\n",
" global test_labels\n",
"\n",
" predictions = run_tflite_model(tflite_file, [test_image_index])\n",
"\n",
" plt.imshow(test_images[test_image_index])\n",
" template = model_type + \" Model \\n True:{true}, Predicted:{predict}\"\n",
" _ = plt.title(template.format(true= str(test_labels[test_image_index]), predict=str(predictions[0])))\n",
" plt.grid(False)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "A5OTJ_6Vcslt"
},
"source": [
"Now test the float model:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "iTK0x980coto"
},
"outputs": [],
"source": [
"test_model(tflite_model_file, test_image_index, model_type=\"Float\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "o3N6-UGl1dfE"
},
"source": [
"And test the quantized model:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "rc1i9umMcp0t"
},
"outputs": [],
"source": [
"test_model(tflite_model_quant_file, test_image_index, model_type=\"Quantized\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "LwN7uIdCd8Gw"
},
"source": [
"### Evaluate the models on all images"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "RFKOD4DG8XmU"
},
"source": [
"Now let's run both models using all the test images we loaded at the beginning of this tutorial:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "05aeAuWjvjPx"
},
"outputs": [],
"source": [
"# Helper function to evaluate a TFLite model on all images\n",
"def evaluate_model(tflite_file, model_type):\n",
" global test_images\n",
" global test_labels\n",
"\n",
" test_image_indices = range(test_images.shape[0])\n",
" predictions = run_tflite_model(tflite_file, test_image_indices)\n",
"\n",
" accuracy = (np.sum(test_labels== predictions) * 100) / len(test_images)\n",
"\n",
" print('%s model accuracy is %.4f%% (Number of test samples=%d)' % (\n",
" model_type, accuracy, len(test_images)))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "xnFilQpBuMh5"
},
"source": [
"Evaluate the float model:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "T5mWkSbMcU5z"
},
"outputs": [],
"source": [
"evaluate_model(tflite_model_file, model_type=\"Float\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Km3cY9ry8ZlG"
},
"source": [
"Evaluate the quantized model:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "-9cnwiPp6EGm"
},
"outputs": [],
"source": [
"evaluate_model(tflite_model_quant_file, model_type=\"Quantized\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "L7lfxkor8pgv"
},
"source": [
"So you now have an integer quantized a model with almost no difference in the accuracy, compared to the float model.\n",
"\n",
"To learn more about other quantization strategies, read about [TensorFlow Lite model optimization](https://www.tensorflow.org/lite/performance/model_optimization)."
]
}
],
"metadata": {
"colab": {
"collapsed_sections": [],
"name": "post_training_integer_quant.ipynb",
"provenance": [],
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
tensorflow/lite/g3doc/performance/post_training_integer_quant_16x8.ipynb
+591
View File
@@ -0,0 +1,591 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "c8Cx-rUMVX25"
},
"source": [
"##### Copyright 2024 The AI Edge Authors."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "I9sUhVL_VZNO"
},
"outputs": [],
"source": [
"#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6Y8E0lw5eYWm"
},
"source": [
"# Post-training integer quantization with int16 activations"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "CGuqeuPSVNo-"
},
"source": [
"<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
" <td>\n",
" <a target=\"_blank\" href=\"https://www.tensorflow.org/lite/performance/post_training_integer_quant_16x8\"><img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" />View on TensorFlow.org</a>\n",
" </td>\n",
" <td>\n",
" <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/performance/post_training_integer_quant_16x8.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
" </td>\n",
" <td>\n",
" <a target=\"_blank\" href=\"https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/performance/post_training_integer_quant_16x8.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://storage.googleapis.com/tensorflow_docs/tensorflow/tensorflow/lite/g3doc/performance/post_training_integer_quant_16x8.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\" />Download notebook</a>\n",
" </td>\n",
"</table>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "BTC1rDAuei_1"
},
"source": [
"## Overview\n",
"\n",
"[TensorFlow Lite](https://www.tensorflow.org/lite/) now supports\n",
"converting activations to 16-bit integer values and weights to 8-bit integer values during model conversion from TensorFlow to TensorFlow Lite's flat buffer format. We refer to this mode as the \"16x8 quantization mode\". This mode can improve accuracy of the quantized model significantly, when activations are sensitive to the quantization, while still achieving almost 3-4x reduction in model size. Moreover, this fully quantized model can be consumed by integer-only hardware accelerators. \n",
"\n",
"Some examples of models that benefit from this mode of the post-training quantization include: \n",
"* super-resolution, \n",
"* audio signal processing such\n",
"as noise cancelling and beamforming, \n",
"* image de-noising, \n",
"* HDR reconstruction\n",
"from a single image\n",
"\n",
"In this tutorial, you train an MNIST model from scratch, check its accuracy in TensorFlow, and then convert the model into a Tensorflow Lite flatbuffer using this mode. At the end you check the accuracy of the converted model and compare it to the original float32 model. Note that this example demonstrates the usage of this mode and doesn't show benefits over other available quantization techniques in TensorFlow Lite."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2XsEP17Zelz9"
},
"source": [
"## Build an MNIST model"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dDqqUIZjZjac"
},
"source": [
"### Setup"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "gyqAw1M9lyab"
},
"outputs": [],
"source": [
"import logging\n",
"logging.getLogger(\"tensorflow\").setLevel(logging.DEBUG)\n",
"\n",
"import tensorflow as tf\n",
"from tensorflow import keras\n",
"import numpy as np\n",
"import pathlib"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "srTSFKjn1tMp"
},
"source": [
"Check that the 16x8 quantization mode is available "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "c6nb7OPlXs_3"
},
"outputs": [],
"source": [
"tf.lite.OpsSet.EXPERIMENTAL_TFLITE_BUILTINS_ACTIVATIONS_INT16_WEIGHTS_INT8"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "eQ6Q0qqKZogR"
},
"source": [
"### Train and export the model"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "hWSAjQWagIHl"
},
"outputs": [],
"source": [
"# Load MNIST dataset\n",
"mnist = keras.datasets.mnist\n",
"(train_images, train_labels), (test_images, test_labels) = mnist.load_data()\n",
"\n",
"# Normalize the input image so that each pixel value is between 0 to 1.\n",
"train_images = train_images / 255.0\n",
"test_images = test_images / 255.0\n",
"\n",
"# Define the model architecture\n",
"model = keras.Sequential([\n",
" keras.layers.InputLayer(input_shape=(28, 28)),\n",
" keras.layers.Reshape(target_shape=(28, 28, 1)),\n",
" keras.layers.Conv2D(filters=12, kernel_size=(3, 3), activation=tf.nn.relu),\n",
" keras.layers.MaxPooling2D(pool_size=(2, 2)),\n",
" keras.layers.Flatten(),\n",
" keras.layers.Dense(10)\n",
"])\n",
"\n",
"# Train the digit classification model\n",
"model.compile(optimizer='adam',\n",
" loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
" metrics=['accuracy'])\n",
"model.fit(\n",
" train_images,\n",
" train_labels,\n",
" epochs=1,\n",
" validation_data=(test_images, test_labels)\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5NMaNZQCkW9X"
},
"source": [
"For the example, you trained the model for just a single epoch, so it only trains to ~96% accuracy."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "xl8_fzVAZwOh"
},
"source": [
"### Convert to a TensorFlow Lite model\n",
"\n",
"Using the TensorFlow Lite [Converter](https://www.tensorflow.org/lite/models/convert), you can now convert the trained model into a TensorFlow Lite model.\n",
"\n",
"Now, convert the model using `TFliteConverter` into default float32 format:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "_i8B2nDZmAgQ"
},
"outputs": [],
"source": [
"converter = tf.lite.TFLiteConverter.from_keras_model(model)\n",
"tflite_model = converter.convert()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "F2o2ZfF0aiCx"
},
"source": [
"Write it out to a `.tflite` file:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "vptWZq2xnclo"
},
"outputs": [],
"source": [
"tflite_models_dir = pathlib.Path(\"/tmp/mnist_tflite_models/\")\n",
"tflite_models_dir.mkdir(exist_ok=True, parents=True)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Ie9pQaQrn5ue"
},
"outputs": [],
"source": [
"tflite_model_file = tflite_models_dir/\"mnist_model.tflite\"\n",
"tflite_model_file.write_bytes(tflite_model)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7BONhYtYocQY"
},
"source": [
"To instead quantize the model to 16x8 quantization mode, first set the `optimizations` flag to use default optimizations. Then specify that 16x8 quantization mode is the required supported operation in the target specification:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "HEZ6ET1AHAS3"
},
"outputs": [],
"source": [
"converter.optimizations = [tf.lite.Optimize.DEFAULT]\n",
"converter.target_spec.supported_ops = [tf.lite.OpsSet.EXPERIMENTAL_TFLITE_BUILTINS_ACTIVATIONS_INT16_WEIGHTS_INT8]"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "zLxQwZq9CpN7"
},
"source": [
"As in the case of int8 post-training quantization, it is possible to produce a fully integer quantized model by setting converter options `inference_input(output)_type` to tf.int16."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "yZekFJC5-fOG"
},
"source": [
"Set the calibration data:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Y3a6XFqvHbYM"
},
"outputs": [],
"source": [
"mnist_train, _ = tf.keras.datasets.mnist.load_data()\n",
"images = tf.cast(mnist_train[0], tf.float32) / 255.0\n",
"mnist_ds = tf.data.Dataset.from_tensor_slices((images)).batch(1)\n",
"def representative_data_gen():\n",
" for input_value in mnist_ds.take(100):\n",
" # Model has only one input so each data point has one element.\n",
" yield [input_value]\n",
"converter.representative_dataset = representative_data_gen"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "xW84iMYjHd9t"
},
"source": [
"Finally, convert the model as usual. Note, by default the converted model will still use float input and outputs for invocation convenience."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "yuNfl3CoHNK3"
},
"outputs": [],
"source": [
"tflite_16x8_model = converter.convert()\n",
"tflite_model_16x8_file = tflite_models_dir/\"mnist_model_quant_16x8.tflite\"\n",
"tflite_model_16x8_file.write_bytes(tflite_16x8_model)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "PhMmUTl4sbkz"
},
"source": [
"Note how the resulting file is approximately `1/3` the size."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "JExfcfLDscu4"
},
"outputs": [],
"source": [
"!ls -lh {tflite_models_dir}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "L8lQHMp_asCq"
},
"source": [
"## Run the TensorFlow Lite models"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "-5l6-ciItvX6"
},
"source": [
"Run the TensorFlow Lite model using the Python TensorFlow Lite Interpreter."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Ap_jE7QRvhPf"
},
"source": [
"### Load the model into the interpreters"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Jn16Rc23zTss"
},
"outputs": [],
"source": [
"interpreter = tf.lite.Interpreter(model_path=str(tflite_model_file))\n",
"interpreter.allocate_tensors()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "J8Pztk1mvNVL"
},
"outputs": [],
"source": [
"interpreter_16x8 = tf.lite.Interpreter(model_path=str(tflite_model_16x8_file))\n",
"interpreter_16x8.allocate_tensors()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2opUt_JTdyEu"
},
"source": [
"### Test the models on one image"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "AKslvo2kwWac"
},
"outputs": [],
"source": [
"test_image = np.expand_dims(test_images[0], axis=0).astype(np.float32)\n",
"\n",
"input_index = interpreter.get_input_details()[0][\"index\"]\n",
"output_index = interpreter.get_output_details()[0][\"index\"]\n",
"\n",
"interpreter.set_tensor(input_index, test_image)\n",
"interpreter.invoke()\n",
"predictions = interpreter.get_tensor(output_index)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "XZClM2vo3_bm"
},
"outputs": [],
"source": [
"import matplotlib.pylab as plt\n",
"\n",
"plt.imshow(test_images[0])\n",
"template = \"True:{true}, predicted:{predict}\"\n",
"_ = plt.title(template.format(true= str(test_labels[0]),\n",
" predict=str(np.argmax(predictions[0]))))\n",
"plt.grid(False)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "3gwhv4lKbYZ4"
},
"outputs": [],
"source": [
"test_image = np.expand_dims(test_images[0], axis=0).astype(np.float32)\n",
"\n",
"input_index = interpreter_16x8.get_input_details()[0][\"index\"]\n",
"output_index = interpreter_16x8.get_output_details()[0][\"index\"]\n",
"\n",
"interpreter_16x8.set_tensor(input_index, test_image)\n",
"interpreter_16x8.invoke()\n",
"predictions = interpreter_16x8.get_tensor(output_index)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "CIH7G_MwbY2x"
},
"outputs": [],
"source": [
"plt.imshow(test_images[0])\n",
"template = \"True:{true}, predicted:{predict}\"\n",
"_ = plt.title(template.format(true= str(test_labels[0]),\n",
" predict=str(np.argmax(predictions[0]))))\n",
"plt.grid(False)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "LwN7uIdCd8Gw"
},
"source": [
"### Evaluate the models"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "05aeAuWjvjPx"
},
"outputs": [],
"source": [
"# A helper function to evaluate the TF Lite model using \"test\" dataset.\n",
"def evaluate_model(interpreter):\n",
" input_index = interpreter.get_input_details()[0][\"index\"]\n",
" output_index = interpreter.get_output_details()[0][\"index\"]\n",
"\n",
" # Run predictions on every image in the \"test\" dataset.\n",
" prediction_digits = []\n",
" for test_image in test_images:\n",
" # Pre-processing: add batch dimension and convert to float32 to match with\n",
" # the model's input data format.\n",
" test_image = np.expand_dims(test_image, axis=0).astype(np.float32)\n",
" interpreter.set_tensor(input_index, test_image)\n",
"\n",
" # Run inference.\n",
" interpreter.invoke()\n",
"\n",
" # Post-processing: remove batch dimension and find the digit with highest\n",
" # probability.\n",
" output = interpreter.tensor(output_index)\n",
" digit = np.argmax(output()[0])\n",
" prediction_digits.append(digit)\n",
"\n",
" # Compare prediction results with ground truth labels to calculate accuracy.\n",
" accurate_count = 0\n",
" for index in range(len(prediction_digits)):\n",
" if prediction_digits[index] == test_labels[index]:\n",
" accurate_count += 1\n",
" accuracy = accurate_count * 1.0 / len(prediction_digits)\n",
"\n",
" return accuracy"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "T5mWkSbMcU5z"
},
"outputs": [],
"source": [
"print(evaluate_model(interpreter))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Km3cY9ry8ZlG"
},
"source": [
"Repeat the evaluation on the 16x8 quantized model:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "-9cnwiPp6EGm"
},
"outputs": [],
"source": [
"# NOTE: This quantization mode is an experimental post-training mode,\n",
"# it does not have any optimized kernels implementations or\n",
"# specialized machine learning hardware accelerators. Therefore,\n",
"# it could be slower than the float interpreter.\n",
"print(evaluate_model(interpreter_16x8))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "L7lfxkor8pgv"
},
"source": [
"In this example, you have quantized a model to 16x8 with no difference in the accuracy, but with the 3x reduced size.\n"
]
}
],
"metadata": {
"colab": {
"collapsed_sections": [],
"name": "post_training_integer_quant_16x8.ipynb",
"provenance": [],
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
tensorflow/lite/g3doc/performance/post_training_quant.ipynb
+572
View File
@@ -0,0 +1,572 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "_-GR0EDHM1SO"
},
"source": [
"##### Copyright 2024 The AI Edge Authors."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "R3yYtBPkM2qZ"
},
"outputs": [],
"source": [
"#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6Y8E0lw5eYWm"
},
"source": [
"# Post-training dynamic range quantization"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "CIGrZZPTZVeO"
},
"source": [
"<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
" <td>\n",
" <a target=\"_blank\" href=\"https://www.tensorflow.org/lite/performance/post_training_quant\"><img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" />View on TensorFlow.org</a>\n",
" </td>\n",
" <td>\n",
" <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/performance/post_training_quant.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
" </td>\n",
" <td>\n",
" <a target=\"_blank\" href=\"https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/performance/post_training_quant.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://storage.googleapis.com/tensorflow_docs/tensorflow/tensorflow/lite/g3doc/performance/post_training_quant.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\" />Download notebook</a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://tfhub.dev/google/imagenet/resnet_v2_101/classification/4\"><img src=\"https://www.tensorflow.org/images/hub_logo_32px.png\" />See TF Hub model</a>\n",
" </td>\n",
"</table>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "BTC1rDAuei_1"
},
"source": [
"## Overview\n",
"\n",
"[TensorFlow Lite](https://www.tensorflow.org/lite/) now supports\n",
"converting weights to 8-bit precision as part of model conversion from\n",
"tensorflow graphdefs to TensorFlow Lite's flat buffer format. Dynamic range quantization achieves a 4x reduction in the model size. In addition, TFLite supports on-the-fly quantization and dequantization of activations to allow for:\n",
"\n",
"1. Using quantized kernels for faster implementation when available.\n",
"2. Mixing of floating-point kernels with quantized kernels for different parts\n",
" of the graph.\n",
"\n",
"The activations are always stored in floating point. For ops that\n",
"support quantized kernels, the activations are quantized to 8 bits of precision\n",
"dynamically prior to processing and are de-quantized to float precision after\n",
"processing. Depending on the model being converted, this can give a speedup over\n",
"pure floating point computation.\n",
"\n",
"In contrast to\n",
"[quantization aware training](https://github.com/tensorflow/tensorflow/tree/r1.14/tensorflow/contrib/quantize)\n",
", the weights are quantized post training and the activations are quantized dynamically \n",
"at inference in this method.\n",
"Therefore, the model weights are not retrained to compensate for quantization\n",
"induced errors. It is important to check the accuracy of the quantized model to\n",
"ensure that the degradation is acceptable.\n",
"\n",
"This tutorial trains an MNIST model from scratch, checks its accuracy in\n",
"TensorFlow, and then converts the model into a Tensorflow Lite flatbuffer\n",
"with dynamic range quantization. Finally, it checks the\n",
"accuracy of the converted model and compares it to the original float model."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2XsEP17Zelz9"
},
"source": [
"## Build an MNIST model"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dDqqUIZjZjac"
},
"source": [
"### Setup"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "gyqAw1M9lyab"
},
"outputs": [],
"source": [
"import logging\n",
"logging.getLogger(\"tensorflow\").setLevel(logging.DEBUG)\n",
"\n",
"import tensorflow as tf\n",
"from tensorflow import keras\n",
"import numpy as np\n",
"import pathlib"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "eQ6Q0qqKZogR"
},
"source": [
"### Train a TensorFlow model"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "hWSAjQWagIHl"
},
"outputs": [],
"source": [
"# Load MNIST dataset\n",
"mnist = keras.datasets.mnist\n",
"(train_images, train_labels), (test_images, test_labels) = mnist.load_data()\n",
"\n",
"# Normalize the input image so that each pixel value is between 0 to 1.\n",
"train_images = train_images / 255.0\n",
"test_images = test_images / 255.0\n",
"\n",
"# Define the model architecture\n",
"model = keras.Sequential([\n",
" keras.layers.InputLayer(input_shape=(28, 28)),\n",
" keras.layers.Reshape(target_shape=(28, 28, 1)),\n",
" keras.layers.Conv2D(filters=12, kernel_size=(3, 3), activation=tf.nn.relu),\n",
" keras.layers.MaxPooling2D(pool_size=(2, 2)),\n",
" keras.layers.Flatten(),\n",
" keras.layers.Dense(10)\n",
"])\n",
"\n",
"# Train the digit classification model\n",
"model.compile(optimizer='adam',\n",
" loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
" metrics=['accuracy'])\n",
"model.fit(\n",
" train_images,\n",
" train_labels,\n",
" epochs=1,\n",
" validation_data=(test_images, test_labels)\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5NMaNZQCkW9X"
},
"source": [
"For example, since you trained the model for just a single epoch, so it only trains to ~96% accuracy.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "xl8_fzVAZwOh"
},
"source": [
"### Convert to a TensorFlow Lite model\n",
"\n",
"Using the TensorFlow Lite [Converter](https://www.tensorflow.org/lite/models/convert), you can now convert the trained model into a TensorFlow Lite model.\n",
"\n",
"Now load the model using the `TFLiteConverter`:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "_i8B2nDZmAgQ"
},
"outputs": [],
"source": [
"converter = tf.lite.TFLiteConverter.from_keras_model(model)\n",
"tflite_model = converter.convert()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "F2o2ZfF0aiCx"
},
"source": [
"Write it out to a tflite file:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "vptWZq2xnclo"
},
"outputs": [],
"source": [
"tflite_models_dir = pathlib.Path(\"/tmp/mnist_tflite_models/\")\n",
"tflite_models_dir.mkdir(exist_ok=True, parents=True)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Ie9pQaQrn5ue"
},
"outputs": [],
"source": [
"tflite_model_file = tflite_models_dir/\"mnist_model.tflite\"\n",
"tflite_model_file.write_bytes(tflite_model)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7BONhYtYocQY"
},
"source": [
"To quantize the model on export, set the `optimizations` flag to optimize for size:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "g8PUvLWDlmmz"
},
"outputs": [],
"source": [
"converter.optimizations = [tf.lite.Optimize.DEFAULT]\n",
"tflite_quant_model = converter.convert()\n",
"tflite_model_quant_file = tflite_models_dir/\"mnist_model_quant.tflite\"\n",
"tflite_model_quant_file.write_bytes(tflite_quant_model)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "PhMmUTl4sbkz"
},
"source": [
"Note how the resulting file, is approximately `1/4` the size."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "JExfcfLDscu4"
},
"outputs": [],
"source": [
"!ls -lh {tflite_models_dir}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "L8lQHMp_asCq"
},
"source": [
"## Run the TFLite models\n",
"\n",
"Run the TensorFlow Lite model using the Python TensorFlow Lite\n",
"Interpreter.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Ap_jE7QRvhPf"
},
"source": [
"### Load the model into an interpreter"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Jn16Rc23zTss"
},
"outputs": [],
"source": [
"interpreter = tf.lite.Interpreter(model_path=str(tflite_model_file))\n",
"interpreter.allocate_tensors()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "J8Pztk1mvNVL"
},
"outputs": [],
"source": [
"interpreter_quant = tf.lite.Interpreter(model_path=str(tflite_model_quant_file))\n",
"interpreter_quant.allocate_tensors()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2opUt_JTdyEu"
},
"source": [
"### Test the model on one image"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "AKslvo2kwWac"
},
"outputs": [],
"source": [
"test_image = np.expand_dims(test_images[0], axis=0).astype(np.float32)\n",
"\n",
"input_index = interpreter.get_input_details()[0][\"index\"]\n",
"output_index = interpreter.get_output_details()[0][\"index\"]\n",
"\n",
"interpreter.set_tensor(input_index, test_image)\n",
"interpreter.invoke()\n",
"predictions = interpreter.get_tensor(output_index)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "XZClM2vo3_bm"
},
"outputs": [],
"source": [
"import matplotlib.pylab as plt\n",
"\n",
"plt.imshow(test_images[0])\n",
"template = \"True:{true}, predicted:{predict}\"\n",
"_ = plt.title(template.format(true= str(test_labels[0]),\n",
" predict=str(np.argmax(predictions[0]))))\n",
"plt.grid(False)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "LwN7uIdCd8Gw"
},
"source": [
"### Evaluate the models"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "05aeAuWjvjPx"
},
"outputs": [],
"source": [
"# A helper function to evaluate the TF Lite model using \"test\" dataset.\n",
"def evaluate_model(interpreter):\n",
" input_index = interpreter.get_input_details()[0][\"index\"]\n",
" output_index = interpreter.get_output_details()[0][\"index\"]\n",
"\n",
" # Run predictions on every image in the \"test\" dataset.\n",
" prediction_digits = []\n",
" for test_image in test_images:\n",
" # Pre-processing: add batch dimension and convert to float32 to match with\n",
" # the model's input data format.\n",
" test_image = np.expand_dims(test_image, axis=0).astype(np.float32)\n",
" interpreter.set_tensor(input_index, test_image)\n",
"\n",
" # Run inference.\n",
" interpreter.invoke()\n",
"\n",
" # Post-processing: remove batch dimension and find the digit with the highest\n",
" # probability.\n",
" output = interpreter.tensor(output_index)\n",
" digit = np.argmax(output()[0])\n",
" prediction_digits.append(digit)\n",
"\n",
" # Compare prediction results with ground truth labels to calculate accuracy.\n",
" accurate_count = 0\n",
" for index in range(len(prediction_digits)):\n",
" if prediction_digits[index] == test_labels[index]:\n",
" accurate_count += 1\n",
" accuracy = accurate_count * 1.0 / len(prediction_digits)\n",
"\n",
" return accuracy"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "DqXBnDfJ7qxL"
},
"outputs": [],
"source": [
"print(evaluate_model(interpreter))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Km3cY9ry8ZlG"
},
"source": [
"Repeat the evaluation on the dynamic range quantized model to obtain:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "-9cnwiPp6EGm"
},
"outputs": [],
"source": [
"print(evaluate_model(interpreter_quant))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "L7lfxkor8pgv"
},
"source": [
"In this example, the compressed model has no difference in the accuracy."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "M0o1FtmWeKZm"
},
"source": [
"## Optimizing an existing model\n",
"\n",
"Resnets with pre-activation layers (Resnet-v2) are widely used for vision applications.\n",
" Pre-trained frozen graph for resnet-v2-101 is available on\n",
" [Tensorflow Hub](https://tfhub.dev/google/imagenet/resnet_v2_101/classification/4).\n",
"\n",
"You can convert the frozen graph to a TensorFLow Lite flatbuffer with quantization by:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "jrXZxSJiJfYN"
},
"outputs": [],
"source": [
"import tensorflow_hub as hub\n",
"\n",
"resnet_v2_101 = tf.keras.Sequential([\n",
" keras.layers.InputLayer(input_shape=(224, 224, 3)),\n",
" hub.KerasLayer(\"https://tfhub.dev/google/imagenet/resnet_v2_101/classification/4\")\n",
"])\n",
"\n",
"converter = tf.lite.TFLiteConverter.from_keras_model(resnet_v2_101)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "LwnV4KxwVEoG"
},
"outputs": [],
"source": [
"# Convert to TF Lite without quantization\n",
"resnet_tflite_file = tflite_models_dir/\"resnet_v2_101.tflite\"\n",
"resnet_tflite_file.write_bytes(converter.convert())"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "2qkZD0VoVExe"
},
"outputs": [],
"source": [
"# Convert to TF Lite with quantization\n",
"converter.optimizations = [tf.lite.Optimize.DEFAULT]\n",
"resnet_quantized_tflite_file = tflite_models_dir/\"resnet_v2_101_quantized.tflite\"\n",
"resnet_quantized_tflite_file.write_bytes(converter.convert())"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "vhOjeg1x9Knp"
},
"outputs": [],
"source": [
"!ls -lh {tflite_models_dir}/*.tflite"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "qqHLaqFMCjRZ"
},
"source": [
"The model size was reduced from 171 MB to 43 MB.\n",
"The accuracy of this model on imagenet can be evaluated using the scripts provided for [TFLite accuracy measurement](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/tools/evaluation/tasks/imagenet_image_classification).\n",
"\n",
"The optimized model top-1 accuracy is 76.8, the same as the floating point model."
]
}
],
"metadata": {
"colab": {
"collapsed_sections": [],
"name": "post_training_quant.ipynb",
"provenance": [],
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
tensorflow/lite/g3doc/performance/post_training_quantization.md
+284
View File
@@ -0,0 +1,284 @@
# Post-training quantization
Post-training quantization is a conversion technique that can reduce model size
while also improving CPU and hardware accelerator latency, with little
degradation in model accuracy. You can quantize an already-trained float
TensorFlow model when you convert it to TensorFlow Lite format using the
[TensorFlow Lite Converter](../models/convert/).
Note: The procedures on this page require TensorFlow 1.15 or higher.
### Optimization Methods
There are several post-training quantization options to choose from. Here is a
summary table of the choices and the benefits they provide:
| Technique | Benefits | Hardware |
| -------------------- | ------------------------- | ---------------- |
| Dynamic range | 4x smaller, 2x-3x speedup | CPU |
: quantization : : :
| Full integer | 4x smaller, 3x+ speedup | CPU, Edge TPU, |
: quantization : : Microcontrollers :
| Float16 quantization | 2x smaller, GPU | CPU, GPU |
: : acceleration : :
The following decision tree can help determine which post-training quantization
method is best for your use case:
![post-training optimization options](images/optimization.jpg)
### Dynamic range quantization
Dynamic range quantization is a recommended starting point because it provides
reduced memory usage and faster computation without you having to provide a
representative dataset for calibration. This type of quantization, statically
quantizes only the weights from floating point to integer at conversion time,
which provides 8-bits of precision:
<pre>
import tensorflow as tf
converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
<b>converter.optimizations = [tf.lite.Optimize.DEFAULT]</b>
tflite_quant_model = converter.convert()
</pre>
To further reduce latency during inference, "dynamic-range" operators
dynamically quantize activations based on their range to 8-bits and perform
computations with 8-bit weights and activations. This optimization provides
latencies close to fully fixed-point inferences. However, the outputs are still
stored using floating point so the increased speed of dynamic-range ops is less
than a full fixed-point computation.
### Full integer quantization
You can get further latency improvements, reductions in peak memory usage, and
compatibility with integer only hardware devices or accelerators by making sure
all model math is integer quantized.
For full integer quantization, you need to calibrate or estimate the range, i.e,
(min, max) of all floating-point tensors in the model. Unlike constant tensors
such as weights and biases, variable tensors such as model input, activations
(outputs of intermediate layers) and model output cannot be calibrated unless we
run a few inference cycles. As a result, the converter requires a representative
dataset to calibrate them. This dataset can be a small subset (around ~100-500
samples) of the training or validation data. Refer to the
`representative_dataset()` function below.
From TensorFlow 2.7 version, you can specify the representative dataset through
a [signature](../guide/signatures.ipynb) as the following example:
<pre>
def representative_dataset():
for data in dataset:
yield {
"image": data.image,
"bias": data.bias,
}
</pre>
If there are more than one signature in the given TensorFlow model, you can
specify the multiple dataset by specifying the signature keys:
<pre>
def representative_dataset():
# Feed data set for the "encode" signature.
for data in encode_signature_dataset:
yield (
"encode", {
"image": data.image,
"bias": data.bias,
}
)
# Feed data set for the "decode" signature.
for data in decode_signature_dataset:
yield (
"decode", {
"image": data.image,
"hint": data.hint,
},
)
</pre>
You can generate the representative dataset by providing an input tensor list:
<pre>
def representative_dataset():
for data in tf.data.Dataset.from_tensor_slices((images)).batch(1).take(100):
yield [tf.dtypes.cast(data, tf.float32)]
</pre>
Since TensorFlow 2.7 version, we recommend using the signature-based approach
over the input tensor list-based approach because the input tensor ordering can
be easily flipped.
For testing purposes, you can use a dummy dataset as follows:
<pre>
def representative_dataset():
for _ in range(100):
data = np.random.rand(1, 244, 244, 3)
yield [data.astype(np.float32)]
</pre>
#### Integer with float fallback (using default float input/output)
In order to fully integer quantize a model, but use float operators when they
don't have an integer implementation (to ensure conversion occurs smoothly), use
the following steps:
<pre>
import tensorflow as tf
converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
<b>converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.representative_dataset = representative_dataset</b>
tflite_quant_model = converter.convert()
</pre>
Note: This `tflite_quant_model` won't be compatible with integer only devices
(such as 8-bit microcontrollers) and accelerators (such as the Coral Edge TPU)
because the input and output still remain float in order to have the same
interface as the original float only model.
#### Integer only
*Creating integer only models is a common use case for
[TensorFlow Lite for Microcontrollers](https://www.tensorflow.org/lite/microcontrollers)
and [Coral Edge TPUs](https://coral.ai/).*
Note: Starting TensorFlow 2.3.0, we support the `inference_input_type` and
`inference_output_type` attributes.
Additionally, to ensure compatibility with integer only devices (such as 8-bit
microcontrollers) and accelerators (such as the Coral Edge TPU), you can enforce
full integer quantization for all ops including the input and output, by using
the following steps:
<pre>
import tensorflow as tf
converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.representative_dataset = representative_dataset
<b>converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]</b>
<b>converter.inference_input_type = tf.int8</b> # or tf.uint8
<b>converter.inference_output_type = tf.int8</b> # or tf.uint8
tflite_quant_model = converter.convert()
</pre>
### Float16 quantization
You can reduce the size of a floating point model by quantizing the weights to
float16, the IEEE standard for 16-bit floating point numbers. To enable float16
quantization of weights, use the following steps:
<pre>
import tensorflow as tf
converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
<b>converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.target_spec.supported_types = [tf.float16]</b>
tflite_quant_model = converter.convert()
</pre>
The advantages of float16 quantization are as follows:
* It reduces model size by up to half (since all weights become half of their
original size).
* It causes minimal loss in accuracy.
* It supports some delegates (e.g. the GPU delegate) which can operate
directly on float16 data, resulting in faster execution than float32
computations.
The disadvantages of float16 quantization are as follows:
* It does not reduce latency as much as a quantization to fixed point math.
* By default, a float16 quantized model will "dequantize" the weights values
to float32 when run on the CPU. (Note that the GPU delegate will not perform
this dequantization, since it can operate on float16 data.)
### Integer only: 16-bit activations with 8-bit weights (experimental)
This is an experimental quantization scheme. It is similar to the "integer only"
scheme, but activations are quantized based on their range to 16-bits, weights
are quantized in 8-bit integer and bias is quantized into 64-bit integer. This
is referred to as 16x8 quantization further.
The main advantage of this quantization is that it can improve accuracy
significantly, but only slightly increase model size.
<pre>
import tensorflow as tf
converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
converter.representative_dataset = representative_dataset
<b>converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.target_spec.supported_ops = [tf.lite.OpsSet.EXPERIMENTAL_TFLITE_BUILTINS_ACTIVATIONS_INT16_WEIGHTS_INT8]</b>
tflite_quant_model = converter.convert()
</pre>
If 16x8 quantization is not supported for some operators in the model,
then the model still can be quantized, but unsupported operators kept in float.
The following option should be added to the target_spec to allow this.
<pre>
import tensorflow as tf
converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
converter.representative_dataset = representative_dataset
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.target_spec.supported_ops = [tf.lite.OpsSet.EXPERIMENTAL_TFLITE_BUILTINS_ACTIVATIONS_INT16_WEIGHTS_INT8,
<b>tf.lite.OpsSet.TFLITE_BUILTINS</b>]
tflite_quant_model = converter.convert()
</pre>
Examples of the use cases where accuracy improvements provided by this
quantization scheme include:
* super-resolution,
* audio signal processing such as noise cancelling and beamforming,
* image de-noising,
* HDR reconstruction from a single image.
The disadvantage of this quantization is:
* Currently inference is noticeably slower than 8-bit full integer due to the
lack of optimized kernel implementation.
* Currently it is incompatible with the existing hardware accelerated TFLite
delegates.
Note: This is an experimental feature.
A tutorial for this quantization mode can be found
[here](post_training_integer_quant_16x8.ipynb).
### Model accuracy
Since weights are quantized post training, there could be an accuracy loss,
particularly for smaller networks. Pre-trained fully quantized models are
provided for specific networks on
[TensorFlow Hub](https://tfhub.dev/s?deployment-format=lite&q=quantized){:.external}.
It is important to check the accuracy of the quantized model to verify that any
degradation in accuracy is within acceptable limits. There are tools to evaluate
[TensorFlow Lite model accuracy](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/tools/evaluation/tasks){:.external}.
Alternatively, if the accuracy drop is too high, consider using
[quantization aware training](https://www.tensorflow.org/model_optimization/guide/quantization/training)
. However, doing so requires modifications during model training to add fake
quantization nodes, whereas the post-training quantization techniques on this
page use an existing pre-trained model.
### Representation for quantized tensors
8-bit quantization approximates floating point values using the following
formula.
$$real\_value = (int8\_value - zero\_point) \times scale$$
The representation has two main parts:
* Per-axis (aka per-channel) or per-tensor weights represented by int8 two’s
complement values in the range [-127, 127] with zero-point equal to 0.
* Per-tensor activations/inputs represented by int8 two’s complement values in
the range [-128, 127], with a zero-point in range [-128, 127].
For a detailed view of our quantization scheme, please see our
[quantization spec](./quantization_spec). Hardware vendors who want to plug
into TensorFlow Lite's delegate interface are encouraged to implement the
quantization scheme described there.
tensorflow/lite/g3doc/performance/quantization_debugger.ipynb
+856
View File
@@ -0,0 +1,856 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "KimMZUVqcJ8_"
},
"source": [
"##### Copyright 2024 The AI Edge Authors."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "BRQ6HQ8zcV5v"
},
"outputs": [],
"source": [
"#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "BlWzg1D9_EhW"
},
"source": [
"# Inspecting Quantization Errors with Quantization Debugger"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "XLoHL19yb-a0"
},
"source": [
"<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
" <td>\n",
" <a target=\"_blank\" href=\"https://www.tensorflow.org/lite/performance/quantization_debugger\"><img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" />View on TensorFlow.org</a>\n",
" </td>\n",
" <td>\n",
" <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/performance/quantization_debugger.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
" </td>\n",
" <td>\n",
" <a target=\"_blank\" href=\"https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/performance/quantization_debugger.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://storage.googleapis.com/tensorflow_docs/tensorflow/tensorflow/lite/g3doc/performance/quantization_debugger.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\" />Download notebook</a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://tfhub.dev/google/imagenet/mobilenet_v3_small_100_224/classification/5\"><img src=\"https://www.tensorflow.org/images/hub_logo_32px.png\" />See TF Hub model</a>\n",
" </td>\n",
"</table>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "MWO_yYDGcGWY"
},
"source": [
"Although full-integer quantization provides improved model size and latency, the\n",
"quantized model won't always work as expected. It's usually expected for the\n",
"model quality (e.g. accuracy, mAP, WER) to be slightly lower than the original\n",
"float model. However, there are cases where the model quality can go below your\n",
"expectations or generates completely wrong results.\n",
"\n",
"When this problem happens, it's tricky and painful to spot the root cause of the\n",
"quantization error, and it's even more difficult to fix the quantization error.\n",
"To assist this model inspection process, a **quantization debugger** can be used\n",
"to identify problematic layers, and **selective quantization** can leave those\n",
"problematic layers in float so that the model accuracy can be recovered at the\n",
"cost of reduced benefit from quantization.\n",
"\n",
"Note: This API is experimental, and there might be breaking changes in the API\n",
"in the course of improvements."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "9kD29R1I_Mn6"
},
"source": [
"## Quantization Debugger\n",
"\n",
"Quantization debugger makes it possible to do quantization quality metric\n",
"analysis in the existing model. Quantization debugger can automate processes for\n",
"running model with a debug dataset, and collecting quantization quality metrics\n",
"for each tensors.\n",
"\n",
"Note: Quantization debugger and selective quantization currently only works for\n",
"full-integer quantization with int8 activations."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "221Qon7G_PmZ"
},
"source": [
"### Prerequisites\n",
"\n",
"If you already have a pipeline to quantize a model, you have all necessary\n",
"pieces to run quantization debugger!\n",
"\n",
"* Model to quantize\n",
"* Representative dataset\n",
"\n",
"In addition to the model and data, you will need to use a data processing framework\n",
"(e.g. pandas, Google Sheets) to analyze the exported results."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "qTEEzJWo_iZ_"
},
"source": [
"### Setup\n",
"\n",
"This section prepares libraries, MobileNet v3 model, and a test dataset of 100\n",
"images."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "l7epUDUP_6qo"
},
"outputs": [],
"source": [
"# Quantization debugger is available from TensorFlow 2.7.0\n",
"!pip uninstall -y tensorflow\n",
"!pip install tf-nightly\n",
"!pip install tensorflow_datasets --upgrade # imagenet_v2 needs latest checksum"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "LLsgiUZe_hIa"
},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"import numpy as np\n",
"import pandas as pd\n",
"import tensorflow as tf\n",
"import tensorflow_datasets as tfds\n",
"import tensorflow_hub as hub"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "veWjO3u32vzz"
},
"outputs": [],
"source": [
"#@title Boilerplates and helpers\n",
"MODEL_URI = 'https://tfhub.dev/google/imagenet/mobilenet_v3_small_100_224/classification/5'\n",
"\n",
"\n",
"def process_image(data):\n",
" data['image'] = tf.image.resize(data['image'], (224, 224)) / 255.0\n",
" return data\n",
"\n",
"\n",
"# Representative dataset\n",
"def representative_dataset(dataset):\n",
"\n",
" def _data_gen():\n",
" for data in dataset.batch(1):\n",
" yield [data['image']]\n",
"\n",
" return _data_gen\n",
"\n",
"\n",
"def eval_tflite(tflite_model, dataset):\n",
" \"\"\"Evaluates tensorflow lite classification model with the given dataset.\"\"\"\n",
" interpreter = tf.lite.Interpreter(model_content=tflite_model)\n",
" interpreter.allocate_tensors()\n",
"\n",
" input_idx = interpreter.get_input_details()[0]['index']\n",
" output_idx = interpreter.get_output_details()[0]['index']\n",
"\n",
" results = []\n",
"\n",
" for data in representative_dataset(dataset)():\n",
" interpreter.set_tensor(input_idx, data[0])\n",
" interpreter.invoke()\n",
" results.append(interpreter.get_tensor(output_idx).flatten())\n",
"\n",
" results = np.array(results)\n",
" gt_labels = np.array(list(dataset.map(lambda data: data['label'] + 1)))\n",
" accuracy = (\n",
" np.sum(np.argsort(results, axis=1)[:, -5:] == gt_labels.reshape(-1, 1)) /\n",
" gt_labels.size)\n",
" print(f'Top-5 accuracy (quantized): {accuracy * 100:.2f}%')\n",
"\n",
"\n",
"model = tf.keras.Sequential([\n",
" tf.keras.layers.Input(shape=(224, 224, 3), batch_size=1),\n",
" hub.KerasLayer(MODEL_URI)\n",
"])\n",
"model.compile(\n",
" loss='sparse_categorical_crossentropy',\n",
" metrics='sparse_top_k_categorical_accuracy')\n",
"model.build([1, 224, 224, 3])\n",
"\n",
"# Prepare dataset with 100 examples\n",
"ds = tfds.load('imagenet_v2', split='test[:1%]')\n",
"ds = ds.map(process_image)\n",
"\n",
"converter = tf.lite.TFLiteConverter.from_keras_model(model)\n",
"converter.representative_dataset = representative_dataset(ds)\n",
"converter.optimizations = [tf.lite.Optimize.DEFAULT]\n",
"quantized_model = converter.convert()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "7mX-R-xK4ADB"
},
"outputs": [],
"source": [
"test_ds = ds.map(lambda data: (data['image'], data['label'] + 1)).batch(16)\n",
"loss, acc = model.evaluate(test_ds)\n",
"print(f'Top-5 accuracy (float): {acc * 100:.2f}%')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Mnp6yBnJSCoh"
},
"outputs": [],
"source": [
"eval_tflite(quantized_model, ds)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Tblkk3cxxpuw"
},
"source": [
"We can see that the original model has a much higher top-5 accuracy for our\n",
"small dataset, while the quantized model has a significant accuracy loss."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dBBcfCQw_Wqd"
},
"source": [
"### Step 1. Debugger preparation\n",
"\n",
"The Easiest way to use the quantization debugger is to provide\n",
"`tf.lite.TFLiteConverter` that you have been using to quantize the model."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "NOByihbD_NZZ"
},
"outputs": [],
"source": [
"converter = tf.lite.TFLiteConverter.from_keras_model(model)\n",
"converter.optimizations = [tf.lite.Optimize.DEFAULT]\n",
"converter.representative_dataset = representative_dataset(ds)\n",
"\n",
"# my_debug_dataset should have the same format as my_representative_dataset\n",
"debugger = tf.lite.experimental.QuantizationDebugger(\n",
" converter=converter, debug_dataset=representative_dataset(ds))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "9vR1IIrmQS9W"
},
"source": [
"### Step 2. Running the debugger and getting the results\n",
"\n",
"When you call `QuantizationDebugger.run()`, the debugger will log differences\n",
"between float tensors and quantized tensors for the same op location, and\n",
"process them with given metrics."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "HsUM54g-_E52"
},
"outputs": [],
"source": [
"debugger.run()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "yQpX_SBUQXvr"
},
"source": [
"The processed metrics can be accessed with\n",
"`QuantizationDebugger.layer_statistics`, or can be dumped to a text file in CSV\n",
"format with `QuantizationDebugger.layer_statistics_dump()`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "U-AGYUAbQUmx"
},
"outputs": [],
"source": [
"RESULTS_FILE = '/tmp/debugger_results.csv'\n",
"with open(RESULTS_FILE, 'w') as f:\n",
" debugger.layer_statistics_dump(f)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "LQzEi6VnQaen"
},
"outputs": [],
"source": [
"!head /tmp/debugger_results.csv"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "4np7VqU-Qfke"
},
"source": [
"For each row in the dump, the op name and index comes first, followed by\n",
"quantization parameters and error metrics (including\n",
"[user-defined error metrics](#custom-metrics), if any). The resulting CSV file\n",
"can be used to pick problematic layers with large quantization error metrics.\n",
"\n",
"With pandas or other data processing libraries, we can inspect detailed\n",
"per-layer error metrics."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "XUcSqYFGQb-f"
},
"outputs": [],
"source": [
"layer_stats = pd.read_csv(RESULTS_FILE)\n",
"layer_stats.head()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7C_oHxWFOV6M"
},
"source": [
"### Step 3. Data analysis\n",
"\n",
"There are various ways to analyze the resulting. First, let's add some useful\n",
"metrics derived from the debugger's outputs. (`scale` means the quantization\n",
"scale factor for each tensor.)\n",
"\n",
"* Range (`256 / scale`)\n",
"* RMSE / scale (`sqrt(mean_squared_error) / scale`)\n",
"\n",
"The `RMSE / scale` is close to `1 / sqrt(12)` (~ 0.289) when quantized\n",
"distribution is similar to the original float distribution, indicating a good\n",
"quantized model. The larger the value is, it's more likely for the layer not\n",
"being quantized well."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "mwviORyJN6e5"
},
"outputs": [],
"source": [
"layer_stats['range'] = 255.0 * layer_stats['scale']\n",
"layer_stats['rmse/scale'] = layer_stats.apply(\n",
" lambda row: np.sqrt(row['mean_squared_error']) / row['scale'], axis=1)\n",
"layer_stats[['op_name', 'range', 'rmse/scale']].head()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "oAAv35CdPvc4"
},
"outputs": [],
"source": [
"plt.figure(figsize=(15, 5))\n",
"ax1 = plt.subplot(121)\n",
"ax1.bar(np.arange(len(layer_stats)), layer_stats['range'])\n",
"ax1.set_ylabel('range')\n",
"ax2 = plt.subplot(122)\n",
"ax2.bar(np.arange(len(layer_stats)), layer_stats['rmse/scale'])\n",
"ax2.set_ylabel('rmse/scale')\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "8pqUQvRUWB3Q"
},
"source": [
"There are many layers with wide ranges, and some layers that have high\n",
"`RMSE/scale` values. Let's get the layers with high error metrics."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "UqFsUX4_Q-cE"
},
"outputs": [],
"source": [
"layer_stats[layer_stats['rmse/scale'] > 0.7][[\n",
" 'op_name', 'range', 'rmse/scale', 'tensor_name'\n",
"]]"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DHeALFTGWl_e"
},
"source": [
"With these layers, you can try selective quantization to see if not quantizing\n",
"those layers improves model quality."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "cvdkjsbwYC6e"
},
"outputs": [],
"source": [
"suspected_layers = list(\n",
" layer_stats[layer_stats['rmse/scale'] > 0.7]['tensor_name'])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "W6RQw9JobOTR"
},
"source": [
"In addition to these, skipping quantization for the first few layers also helps\n",
"improving quantized model's quality."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ikF2bp6NZcXN"
},
"outputs": [],
"source": [
"suspected_layers.extend(list(layer_stats[:5]['tensor_name']))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "1DfT78w6W6Li"
},
"source": [
"## Selective Quantization"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "-pubC-01cGEH"
},
"source": [
"Selective quantization skips quantization for some nodes, so that the\n",
"calculation can happen in the original floating-point domain. When correct\n",
"layers are skipped, we can expect some model quality recovery at the cost of\n",
"increased latency and model size.\n",
"\n",
"However, if you're planning to run quantized models on integer-only accelerators\n",
"(e.g. Hexagon DSP, EdgeTPU), selective quantization would cause fragmentation of\n",
"the model and would result in slower inference latency mainly caused by data\n",
"transfer cost between CPU and those accelerators. To prevent this, you can\n",
"consider running\n",
"[quantization aware training](https://www.tensorflow.org/model_optimization/guide/quantization/training)\n",
"to keep all the layers in integer while preserving the model accuracy."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "EQFBfR7YW-oh"
},
"source": [
"Quantization debugger's option accepts `denylisted_nodes` and `denylisted_ops`\n",
"options for skipping quantization for specific layers, or all instances of\n",
"specific ops. Using the `suspected_layers` we prepared from the previous step, we\n",
"can use a quantization debugger to get a selectively quantized model."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "K5KD0JAEbpsv"
},
"outputs": [],
"source": [
"debug_options = tf.lite.experimental.QuantizationDebugOptions(\n",
" denylisted_nodes=suspected_layers)\n",
"debugger = tf.lite.experimental.QuantizationDebugger(\n",
" converter=converter,\n",
" debug_dataset=representative_dataset(ds),\n",
" debug_options=debug_options)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "pfj9gzv4b7h4"
},
"outputs": [],
"source": [
"selective_quantized_model = debugger.get_nondebug_quantized_model()\n",
"eval_tflite(selective_quantized_model, ds)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "1RkfMYSHdtZy"
},
"source": [
"The accuracy is still lower compared to the original float model, but we have\n",
"notable improvement from the whole quantized model by skipping quantization for\n",
"~10 layers out of 111 layers.\n",
"\n",
"You can also try to not quantize all ops in the same class. For example, to\n",
"skip quantization for all mean ops, you can pass `MEAN` to `denylisted_ops`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ruUoP7SgcLpO"
},
"outputs": [],
"source": [
"debug_options = tf.lite.experimental.QuantizationDebugOptions(\n",
" denylisted_ops=['MEAN'])\n",
"debugger = tf.lite.experimental.QuantizationDebugger(\n",
" converter=converter,\n",
" debug_dataset=representative_dataset(ds),\n",
" debug_options=debug_options)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "oY6kb5g_cO4H"
},
"outputs": [],
"source": [
"selective_quantized_model = debugger.get_nondebug_quantized_model()\n",
"eval_tflite(selective_quantized_model, ds)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "xa8488TeAyx-"
},
"source": [
"With these techniques, we are able to improve the quantized MobileNet V3 model\n",
"accuracy. Next we'll explore advanced techniques to improve the model accuracy\n",
"even more."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ZD75cY9PUb2u"
},
"source": [
"## Advanced usages\n",
"\n",
"With the following features, you can further customize your debugging pipeline."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "aVj9yrQoUfGo"
},
"source": [
"### Custom metrics\n",
"\n",
"By default, the quantization debugger emits five metrics for each float-quant\n",
"difference: tensor size, standard deviation, mean error, max absolute error, and\n",
"mean squared error. You can add more custom metrics by passing them to options.\n",
"For each metrics, the result should be a single float value and the resulting\n",
"metric will be an average of metrics from all examples.\n",
"\n",
"* `layer_debug_metrics`: calculate metric based on diff for each op outputs\n",
" from float and quantized op outputs.\n",
"* `layer_direct_compare_metrics`: rather than getting diff only, this will\n",
" calculate metric based on raw float and quantized tensors, and its\n",
" quantization parameters (scale, zero point)\n",
"* `model_debug_metrics`: **only used when `float_model_(path|content)` is\n",
" passed** to the debugger. In addition to the op-level metrics, the final layer\n",
" output is compared to the reference output from the original float model."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "WqmRQSxoVVwu"
},
"outputs": [],
"source": [
"debug_options = tf.lite.experimental.QuantizationDebugOptions(\n",
" layer_debug_metrics={\n",
" 'mean_abs_error': (lambda diff: np.mean(np.abs(diff)))\n",
" },\n",
" layer_direct_compare_metrics={\n",
" 'correlation':\n",
" lambda f, q, s, zp: (np.corrcoef(f.flatten(),\n",
" (q.flatten() - zp) / s)[0, 1])\n",
" },\n",
" model_debug_metrics={\n",
" 'argmax_accuracy': (lambda f, q: np.mean(np.argmax(f) == np.argmax(q)))\n",
" })\n",
"\n",
"debugger = tf.lite.experimental.QuantizationDebugger(\n",
" converter=converter,\n",
" debug_dataset=representative_dataset(ds),\n",
" debug_options=debug_options)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "PVQ4nEicXz2l"
},
"outputs": [],
"source": [
"debugger.run()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "dfKA90csX9UL"
},
"outputs": [],
"source": [
"CUSTOM_RESULTS_FILE = '/tmp/debugger_results.csv'\n",
"with open(CUSTOM_RESULTS_FILE, 'w') as f:\n",
" debugger.layer_statistics_dump(f)\n",
"\n",
"custom_layer_stats = pd.read_csv(CUSTOM_RESULTS_FILE)\n",
"custom_layer_stats[['op_name', 'mean_abs_error', 'correlation']].tail()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Qqq30oWsZF5b"
},
"source": [
"The result of `model_debug_metrics` can be separately seen from\n",
"`debugger.model_statistics`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "wrXlmzEHYhQ5"
},
"outputs": [],
"source": [
"debugger.model_statistics"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DqJBLIsoUyIg"
},
"source": [
"### Using (internal) mlir_quantize API to access in-depth features\n",
"\n",
"Note: Some features in the following section,\n",
"`TFLiteConverter._experimental_calibrate_only` and `converter.mlir_quantize` are\n",
"experimental internal APIs, and subject to change in a non-backward compatible\n",
"way."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "VJm66Cz-XpeF"
},
"outputs": [],
"source": [
"from tensorflow.lite.python import convert"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2krUVzpiUp3u"
},
"source": [
"#### Whole model verify mode\n",
"\n",
"The default behavior for the debug model generation is per-layer verify. In this\n",
"mode, the input for float and quantize op pair is from the same source (previous\n",
"quantized op). Another mode is whole-model verify, where the float and quantize\n",
"models are separated. This mode would be useful to observe how the error is\n",
"being propagated down the model. To enable, `enable_whole_model_verify=True` to\n",
"`convert.mlir_quantize` while generating the debug model manually."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "5zykINDlVLSg"
},
"outputs": [],
"source": [
"converter = tf.lite.TFLiteConverter.from_keras_model(model)\n",
"converter.representative_dataset = representative_dataset(ds)\n",
"converter.optimizations = [tf.lite.Optimize.DEFAULT]\n",
"converter._experimental_calibrate_only = True\n",
"calibrated_model = converter.convert()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "eqvXlEiFXfSu"
},
"outputs": [],
"source": [
"# Note that enable_numeric_verify and enable_whole_model_verify are set.\n",
"quantized_model = convert.mlir_quantize(\n",
" calibrated_model,\n",
" enable_numeric_verify=True,\n",
" enable_whole_model_verify=True)\n",
"debugger = tf.lite.experimental.QuantizationDebugger(\n",
" quant_debug_model_content=quantized_model,\n",
" debug_dataset=representative_dataset(ds))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "xQ6TFsXQVHMe"
},
"source": [
"#### Selective quantization from an already calibrated model\n",
"\n",
"You can directly call `convert.mlir_quantize` to get the selective quantized\n",
"model from already calibrated model. This would be particularly useful when you\n",
"want to calibrate the model once, and experiment with various denylist\n",
"combinations."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ZCS-Fa9lbdc0"
},
"outputs": [],
"source": [
"selective_quantized_model = convert.mlir_quantize(\n",
" calibrated_model, denylisted_nodes=suspected_layers)\n",
"eval_tflite(selective_quantized_model, ds)"
]
}
],
"metadata": {
"colab": {
"collapsed_sections": [
"Eq_8T2oauIED"
],
"name": "quantization_debugger.ipynb",
"provenance": [],
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
tensorflow/lite/g3doc/performance/quantization_spec.md
+548
View File
@@ -0,0 +1,548 @@
# TensorFlow Lite 8-bit quantization specification
The following document outlines the specification for TensorFlow Lite's 8-bit
quantization scheme. This is intended to assist hardware developers in providing
hardware support for inference with quantized TensorFlow Lite models.
## Specification summary
We are providing a specification, and we can only provide some guarantees on
behaviour if the spec is followed. We also understand different hardware may
have preferences and restrictions that may cause slight deviations when
implementing the spec that result in implementations that are not bit-exact.
Whereas that may be acceptable in most cases (and we will provide a suite of
tests that to the best of our knowledge include per-operation tolerances that we
gathered from several models), the nature of machine learning (and deep learning
in the most common case) makes it impossible to provide any hard guarantees.
8-bit quantization approximates floating point values using the following
formula.
$$real\_value = (int8\_value - zero\_point) \times scale$$
Per-axis (aka per-channel in Conv ops) or per-tensor weights are represented by
`int8` two’s complement values in the range `[-127, 127]` with zero-point equal
to 0. Per-tensor activations/inputs are represented by `int8` two’s complement
values in the range `[-128, 127]`, with a zero-point in range `[-128, 127]`.
There are other exceptions for particular operations that are documented below.
Note: In the past our quantization tooling used per-tensor, asymmetric, `uint8`
quantization. New tooling, reference kernels, and optimized kernels for 8-bit
quantization will use this spec.
## Signed integer vs unsigned integer
TensorFlow Lite quantization will primarily prioritize tooling and kernels for
`int8` quantization for 8-bit. This is for the convenience of symmetric
quantization being represented by zero-point equal to 0. Additionally many
backends have additional optimizations for `int8xint8` accumulation.
## Per-axis vs per-tensor
Per-tensor quantization means that there will be one scale and/or zero-point per
entire tensor. Per-axis quantization means that there will be one scale and/or
`zero_point` per slice in the `quantized_dimension`. The quantized dimension
specifies the dimension of the Tensor's shape that the scales and zero-points
correspond to. For example, a tensor `t`, with `dims=[4, 3, 2, 1]` with
quantization params: `scale=[1.0, 2.0, 3.0]`, `zero_point=[1, 2, 3]`,
`quantization_dimension=1` will be quantized across the second dimension of `t`:
t[:, 0, :, :] will have scale[0]=1.0, zero_point[0]=1
t[:, 1, :, :] will have scale[1]=2.0, zero_point[1]=2
t[:, 2, :, :] will have scale[2]=3.0, zero_point[2]=3
Often, the `quantized_dimension` is the `output_channel` of the weights of
convolutions, but in theory it can be the dimension that corresponds to each
dot-product in the kernel implementation, allowing more quantization granularity
without performance implications. This has large improvements to accuracy.
TFLite has per-axis support for a growing number of operations. At the time of
this document, support exists for Conv2d and DepthwiseConv2d.
## Symmetric vs asymmetric
Activations are asymmetric: they can have their zero-point anywhere within the
signed `int8` range `[-128, 127]`. Many activations are asymmetric in nature and
a zero-point is an relatively inexpensive way to effectively get up to an extra
binary bit of precision. Since activations are only multiplied by constant
weights, the constant zero-point value can be optimized pretty heavily.
Weights are symmetric: forced to have zero-point equal to 0. Weight values are
multiplied by dynamic input and activation values. This means that there is an
unavoidable runtime cost of multiplying the zero-point of the weight with the
activation value. By enforcing that zero-point is 0 we can avoid this cost.
Explanation of the math: this is similar to section 2.3 in
[arXiv:1712.05877](https://arxiv.org/abs/1712.05877), except for the difference
that we allow the scale values to be per-axis. This generalizes readily, as
follows:
$A$ is a $m \times n$ matrix of quantized activations. <br />
$B$ is a $n \times p$ matrix of quantized weights. <br />
Consider multiplying the $j$th row of $A$, $a_j$ by the $k$th column of
$B$, $b_k$, both of length $n$. The quantized integer values and
zero-points values are $q_a$, $z_a$ and $q_b$, $z_b$ respectively.
$$a_j \cdot b_k = \sum_{i=0}^{n} a_{j}^{(i)} b_{k}^{(i)} =
\sum_{i=0}^{n} (q_{a}^{(i)} - z_a) (q_{b}^{(i)} - z_b) =
\sum_{i=0}^{n} q_{a}^{(i)} q_{b}^{(i)} - \sum_{i=0}^{n} q_{a}^{(i)} z_b -
\sum_{i=0}^{n} q_{b}^{(i)} z_a + \sum_{i=0}^{n} z_a z_b$$
<!-- Don't change these `\\(` `\\)` to `$`. mathjax fails here with `$`-->
The \\(\sum_{i=0}^{n} q_{a}^{(i)} q_{b}^{(i)}\\) term is unavoidable since it’s
performing the dot product of the input value and the weight value.
The $$\sum_{i=0}^{n} q_{b}^{(i)} z_a$$ and $$\sum_{i=0}^{n} z_a z_b$$ terms are
made up of constants that remain the same per inference invocation, and thus can
be pre-calculated.
The \\(\sum_{i=0}^{n} q_{a}^{(i)} z_b\\) term needs to be computed every inference
since the activation changes every inference. By enforcing weights to be
symmetric we can remove the cost of this term.
## int8 quantized operator specifications
Below we describe the quantization requirements for our int8 tflite kernels:
```
ADD
Input 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
Input 1:
data_type : int8
range : [-128, 127]
granularity: per-tensor
Output 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
AVERAGE_POOL_2D
Input 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
Output 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
restriction: Input and outputs must all have same scale/zero_point
CONCATENATION
Input ...:
data_type : int8
range : [-128, 127]
granularity: per-tensor
Output 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
restriction: Input and outputs must all have same scale/zero_point
CONV_2D
Input 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
Input 1 (Weight):
data_type : int8
range : [-127, 127]
granularity: per-axis (dim = 0)
restriction: zero_point = 0
Input 2 (Bias):
data_type : int32
range : [int32_min, int32_max]
granularity: per-axis
restriction: (scale, zero_point) = (input0_scale * input1_scale[...], 0)
Output 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
DEPTHWISE_CONV_2D
Input 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
Input 1 (Weight):
data_type : int8
range : [-127, 127]
granularity: per-axis (dim = 3)
restriction: zero_point = 0
Input 2 (Bias):
data_type : int32
range : [int32_min, int32_max]
granularity: per-axis
restriction: (scale, zero_point) = (input0_scale * input1_scale[...], 0)
Output 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
FULLY_CONNECTED
Input 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
Input 1 (Weight):
data_type : int8
range : [-127, 127]
granularity: per-axis (dim = 0)
restriction: zero_point = 0
Input 2 (Bias):
data_type : int32
range : [int32_min, int32_max]
granularity: per-tensor
restriction: (scale, zero_point) = (input0_scale * input1_scale[...], 0)
Output 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
L2_NORMALIZATION
Input 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
Output 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
restriction: (scale, zero_point) = (1.0 / 128.0, 0)
LOGISTIC
Input 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
Output 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
restriction: (scale, zero_point) = (1.0 / 256.0, -128)
MAX_POOL_2D
Input 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
Output 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
restriction: Input and outputs must all have same scale/zero_point
MUL
Input 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
Input 1:
data_type : int8
range : [-128, 127]
granularity: per-tensor
Output 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
RESHAPE
Input 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
Output 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
restriction: Input and outputs must all have same scale/zero_point
RESIZE_BILINEAR
Input 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
Output 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
restriction: Input and outputs must all have same scale/zero_point
SOFTMAX
Input 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
Output 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
restriction: (scale, zero_point) = (1.0 / 256.0, -128)
SPACE_TO_DEPTH
Input 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
Output 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
restriction: Input and outputs must all have same scale/zero_point
TANH
Input 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
Output 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
restriction: (scale, zero_point) = (1.0 / 128.0, 0)
PAD
Input 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
Output 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
restriction: Input and outputs must all have same scale/zero_point
GATHER
Input 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
Output 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
restriction: Input and outputs must all have same scale/zero_point
BATCH_TO_SPACE_ND
Input 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
Output 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
restriction: Input and outputs must all have same scale/zero_point
SPACE_TO_BATCH_ND
Input 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
Output 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
restriction: Input and outputs must all have same scale/zero_point
TRANSPOSE
Input 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
Output 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
restriction: Input and outputs must all have same scale/zero_point
MEAN
Input 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
Output 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
SUB
Input 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
Input 1:
data_type : int8
range : [-128, 127]
granularity: per-tensor
Output 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
SUM
Input 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
Output 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
SQUEEZE
Input 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
Output 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
restriction: Input and outputs must all have same scale/zero_point
LOG_SOFTMAX
Input 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
Output 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
restriction: (scale, zero_point) = (16.0 / 256.0, 127)
MAXIMUM
Input 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
Output 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
restriction: Input and outputs must all have same scale/zero_point
ARG_MAX
Input 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
MINIMUM
Input 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
Output 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
restriction: Input and outputs must all have same scale/zero_point
LESS
Input 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
Input 1:
data_type : int8
range : [-128, 127]
granularity: per-tensor
PADV2
Input 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
Output 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
restriction: Input and outputs must all have same scale/zero_point
GREATER
Input 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
Input 1:
data_type : int8
range : [-128, 127]
granularity: per-tensor
GREATER_EQUAL
Input 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
Input 1:
data_type : int8
range : [-128, 127]
granularity: per-tensor
LESS_EQUAL
Input 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
Input 1:
data_type : int8
range : [-128, 127]
granularity: per-tensor
SLICE
Input 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
Output 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
restriction: Input and outputs must all have same scale/zero_point
EQUAL
Input 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
Input 1:
data_type : int8
range : [-128, 127]
granularity: per-tensor
NOT_EQUAL
Input 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
Input 1:
data_type : int8
range : [-128, 127]
granularity: per-tensor
SHAPE
Input 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
QUANTIZE (Requantization)
Input 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
Output 0:
data_type : int8
range : [-128, 127]
granularity: per-tensor
```
## References
[arXiv:1712.05877](https://arxiv.org/abs/1712.05877)