wehub/tensorflow--tensorflow
1
0
Fork 0
Code Issues Pull Requests Actions 5 Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution
cffconvert / validate (push) Has been skipped
Details
License Check / license-check (push) Failing after 2s
Details

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 12:14:16 +08:00
commit 8a852e4b4e
36502 changed files with 9277225 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tensorflow/lite/g3doc/android/delegates/gpu.md
+241
View File
@@ -0,0 +1,241 @@
# GPU acceleration delegate with Java/Kotlin Interpreter API
Using graphics processing units (GPUs) to run your machine learning (ML) models
can dramatically improve the performance and the user experience
of your ML-enabled applications. On Android devices, you can enable
[*delegate*](../../performance/delegates) and one of the following APIs:
- Java/Kotlin Interpreter API - this guide
- Task library API - [guide](./gpu_task)
- Native (C/C++) API - [guide](./gpu_native)
This page describes how to enable GPU acceleration for TensorFlow Lite models in
Android apps using the Interpreter API.
For more information about using the GPU delegate for
TensorFlow Lite, including best practices and advanced techniques, see the
[GPU delegates](../../performance/gpu) page.
## Use GPU with TensorFlow Lite with Google Play services
The TensorFlow Lite Java/Kotlin [Interpreter API](https://tensorflow.org/lite/api_docs/java/org/tensorflow/lite/InterpreterApi)
provides a set of general purpose APIs for building a machine learning
applications. This section describes how to use the GPU accelerator delegate
with these APIs with TensorFlow Lite with Google Play services.
[TensorFlow Lite with Google Play services](../play_services) is the recommended
path to use TensorFlow Lite on Android. If your application is targeting devices
not running Google Play, see the
[GPU with Interpreter API and standalone TensorFlow Lite](#standalone)
section.
### Add project dependencies
To enable access to the GPU delegate, add
`com.google.android.gms:play-services-tflite-gpu` to your app's `build.gradle`
file:
```
dependencies {
...
implementation 'com.google.android.gms:play-services-tflite-java:16.4.0'
implementation 'com.google.android.gms:play-services-tflite-gpu:16.4.0'
}
```
### Enable GPU acceleration
Then initialize TensorFlow Lite with Google Play services with the GPU support:
<div>
<devsite-selector>
<section>
<h3>Kotlin</h3>
<p><pre class="prettyprint lang-kotlin">
val useGpuTask = TfLiteGpu.isGpuDelegateAvailable(context)
val interpreterTask = useGpuTask.continueWith { useGpuTask ->
TfLite.initialize(context,
TfLiteInitializationOptions.builder()
.setEnableGpuDelegateSupport(useGpuTask.result)
.build())
}
</pre></p>
</section>
<section>
<h3>Java</h3>
<p><pre class="prettyprint lang-java">
Task<boolean> useGpuTask = TfLiteGpu.isGpuDelegateAvailable(context);
Task<Options> interpreterOptionsTask = useGpuTask.continueWith({ task ->
TfLite.initialize(context,
TfLiteInitializationOptions.builder()
.setEnableGpuDelegateSupport(true)
.build());
});
</pre></p>
</section>
</devsite-selector>
</div>
You can finally initialize the interpreter passing a `GpuDelegateFactory`
through `InterpreterApi.Options`:
<div>
<devsite-selector>
<section>
<h3>Kotlin</h3>
<p><pre class="prettyprint lang-kotlin">
val options = InterpreterApi.Options()
.setRuntime(TfLiteRuntime.FROM_SYSTEM_ONLY)
.addDelegateFactory(GpuDelegateFactory())
val interpreter = InterpreterApi(model, options)
// Run inference
writeToInput(input)
interpreter.run(input, output)
readFromOutput(output)
</pre></p>
</section>
<section>
<h3>Java</h3>
<p><pre class="prettyprint lang-java">
Options options = InterpreterApi.Options()
.setRuntime(TfLiteRuntime.FROM_SYSTEM_ONLY)
.addDelegateFactory(new GpuDelegateFactory());
Interpreter interpreter = new InterpreterApi(model, options);
// Run inference
writeToInput(input);
interpreter.run(input, output);
readFromOutput(output);
</pre></p>
</section>
</devsite-selector>
</div>
Note: The GPU delegate must be created on the same thread that runs it.
Otherwise, you may see the following error, `TfLiteGpuDelegate Invoke:
GpuDelegate must run on the same thread where it was initialized.`
The GPU delegate can also be used with ML model binding in Android Studio.
For more information, see
[Generate model interfaces using metadata](../../inference_with_metadata/codegen#acceleration).
## Use GPU with standalone TensorFlow Lite {:#standalone}
If your application is targets devices which are not running Google Play,
it is possible to bundle the GPU delegate to your application and use it
with the standalone version of TensorFlow Lite.
### Add project dependencies
To enable access to the GPU delegate, add
`org.tensorflow:tensorflow-lite-gpu-delegate-plugin` to your app's `build.gradle`
file:
```
dependencies {
...
implementation 'org.tensorflow:tensorflow-lite'
implementation 'org.tensorflow:tensorflow-lite-gpu-delegate-plugin'
}
```
### Enable GPU acceleration
Then run TensorFlow Lite on GPU with `TfLiteDelegate`. In Java, you can specify
the `GpuDelegate` through `Interpreter.Options`.
<div>
<devsite-selector>
<section>
<h3>Kotlin</h3>
<p><pre class="prettyprint lang-kotlin">
import org.tensorflow.lite.Interpreter
import org.tensorflow.lite.gpu.CompatibilityList
import org.tensorflow.lite.gpu.GpuDelegate
val compatList = CompatibilityList()
val options = Interpreter.Options().apply{
if(compatList.isDelegateSupportedOnThisDevice){
// if the device has a supported GPU, add the GPU delegate
val delegateOptions = compatList.bestOptionsForThisDevice
this.addDelegate(GpuDelegate(delegateOptions))
} else {
// if the GPU is not supported, run on 4 threads
this.setNumThreads(4)
}
}
val interpreter = Interpreter(model, options)
// Run inference
writeToInput(input)
interpreter.run(input, output)
readFromOutput(output)
</pre></p>
</section>
<section>
<h3>Java</h3>
<p><pre class="prettyprint lang-java">
import org.tensorflow.lite.Interpreter;
import org.tensorflow.lite.gpu.CompatibilityList;
import org.tensorflow.lite.gpu.GpuDelegate;
// Initialize interpreter with GPU delegate
Interpreter.Options options = new Interpreter.Options();
CompatibilityList compatList = CompatibilityList();
if(compatList.isDelegateSupportedOnThisDevice()){
// if the device has a supported GPU, add the GPU delegate
GpuDelegate.Options delegateOptions = compatList.getBestOptionsForThisDevice();
GpuDelegate gpuDelegate = new GpuDelegate(delegateOptions);
options.addDelegate(gpuDelegate);
} else {
// if the GPU is not supported, run on 4 threads
options.setNumThreads(4);
}
Interpreter interpreter = new Interpreter(model, options);
// Run inference
writeToInput(input);
interpreter.run(input, output);
readFromOutput(output);
</pre></p>
</section>
</devsite-selector>
</div>
### Quantized models {:#quantized-models}
Android GPU delegate libraries support quantized models by default. You do not
have to make any code changes to use quantized models with the GPU delegate. The
following section explains how to disable quantized support for testing or
experimental purposes.
#### Disable quantized model support
The following code shows how to ***disable*** support for quantized models.
<div>
<devsite-selector>
<section>
<h3>Java</h3>
<p><pre class="prettyprint lang-java">
GpuDelegate delegate = new GpuDelegate(new GpuDelegate.Options().setQuantizedModelsAllowed(false));
Interpreter.Options options = (new Interpreter.Options()).addDelegate(delegate);
</pre></p>
</section>
</devsite-selector>
</div>
For more information about running quantized models with GPU acceleration,
see [GPU delegate](../../performance/gpu#quantized-models) overview.
tensorflow/lite/g3doc/android/delegates/gpu_native.md
+180
View File
@@ -0,0 +1,180 @@
# GPU acceleration delegate with C/C++ API
Using graphics processing units (GPUs) to run your machine learning (ML) models
can dramatically improve the performance and the user experience of your
ML-enabled applications. On Android devices, you can enable GPU-accelerated
execution of your models using a
[*delegate*](https://ai.google.dev/edge/litert/performance/delegates) and one of
the following APIs:
- Interpreter API - [guide](./gpu)
- Task library API - [guide](./gpu_task.md)
- Native (C/C++) API - this guide
This guide covers advanced uses of the GPU delegate for the C API, C++ API, and
use of quantized models. For more information about using the GPU delegate for
TensorFlow Lite, including best practices and advanced techniques, see the
[GPU delegates](https://ai.google.dev/edge/litert/performance/gpu) page.
## Enable GPU acceleration
Use the TensorFlow Lite GPU delegate for Android in C or C++ by creating the
delegate with `TfLiteGpuDelegateV2Create()` and destroying it with
`TfLiteGpuDelegateV2Delete()`, as shown in the following example code:
```c++
// Set up interpreter.
auto model = FlatBufferModel::BuildFromFile(model_path);
if (!model) return false;
ops::builtin::BuiltinOpResolver op_resolver;
InterpreterBuilder builder(*model, op_resolver);
// NEW: Prepare GPU delegate.
auto* delegate = TfLiteGpuDelegateV2Create(/*default options=*/nullptr);
builder.AddDelegate(delegate);
// Build interpreter.
std::unique_ptr<Interpreter> interpreter;
if (builder(&interpreter) != kTfLiteOk) return false;
// Run inference.
WriteToInputTensor(interpreter->typed_input_tensor<float>(0));
if (interpreter->Invoke() != kTfLiteOk) return false;
ReadFromOutputTensor(interpreter->typed_output_tensor<float>(0));
// NEW: Clean up.
TfLiteGpuDelegateV2Delete(delegate);
```
Review the `TfLiteGpuDelegateOptionsV2` object code to build a delegate instance
with custom options. You can initialize the default options with
`TfLiteGpuDelegateOptionsV2Default()` and then modify them as necessary.
The TensorFlow Lite GPU delegate for Android in C or C++ uses the
[Bazel](https://bazel.io) build system. You can build the delegate using the
following command:
```sh
bazel build -c opt --config android_arm64 tensorflow/lite/delegates/gpu:delegate # for static library
bazel build -c opt --config android_arm64 tensorflow/lite/delegates/gpu:libtensorflowlite_gpu_delegate.so # for dynamic library
```
When calling `InterpreterBuilder::operator()` (e.g. `builder(&interpreter)`),
`Interpreter::ModifyGraphWithDelegate()`, or
`Interpreter::Invoke()`, the caller must have an `EGLContext` in the current
thread and `Interpreter::Invoke()` must be called from the same `EGLContext`. If
an `EGLContext` does not exist, the delegate creates one internally, but then
you must ensure that `Interpreter::Invoke()` is always called from the same
thread in which `InterpreterBuilder::operator()` or
`Interpreter::ModifyGraphWithDelegate` was called.
#### With TensorFlow Lite in Google Play Services:
If you are using TensorFlow Lite in Google Play Services
[C API](https://ai.google.dev/edge/litert/android/native), you’ll need to use
the Java/Kotlin API to check if a GPU delegate is available for your device
before initializing the TensorFlow Lite runtime.
Add the GPU delegate gradle dependencies to your application:
```
implementation 'com.google.android.gms:play-services-tflite-gpu:16.4.0'
```
Then, check the GPU availability and initialize TfLiteNative if the check is
successful:
<div>
<devsite-selector>
<section>
<h3>Java</h3>
<pre class="prettyprint">
Task<Void> tfLiteHandleTask =
TfLiteGpu.isGpuDelegateAvailable(this)
.onSuccessTask(gpuAvailable -> {
TfLiteInitializationOptions options =
TfLiteInitializationOptions.builder()
.setEnableGpuDelegateSupport(gpuAvailable).build();
return TfLiteNative.initialize(this, options);
}
);
</pre>
</section>
<section>
<h3>Kotlin</h3>
<pre class="prettyprint">
val tfLiteHandleTask = TfLiteGpu.isGpuDelegateAvailable(this)
.onSuccessTask { gpuAvailable ->
val options = TfLiteInitializationOptions.Builder()
.setEnableGpuDelegateSupport(gpuAvailable)
.build()
TfLiteNative.initialize(this, options)
}
</pre>
</section>
</devsite-selector>
</div>
You also need to update your CMake configuration to include the
`TFLITE_USE_OPAQUE_DELEGATE` compiler flag:
```
add_compile_definitions(TFLITE_USE_OPAQUE_DELEGATE)
```
The [FlatBuffers](https://flatbuffers.dev/) library is used to configure
delegate plugins, so you need to add it to the dependencies of your native code.
You can use the official `CMake` project configuration as follow:
```
target_include_directories(tflite-jni PUBLIC
third_party/headers # flatbuffers
...)
```
You can also just bundle the headers to your app.
Finally to use GPU inference in your C code, create the GPU delegate using
`TFLiteSettings`:
```
#include "flatbuffers/flatbuffers.h"
#include "tensorflow/lite/acceleration/configuration/configuration_generated.h"
flatbuffers::FlatBufferBuilder fbb;
tflite::TFLiteSettingsBuilder builder(fbb);
const tflite::TFLiteSettings* tflite_settings =
flatbuffers::GetTemporaryPointer(fbb, builder.Finish());
const TfLiteOpaqueDelegatePlugin* pluginCApi = TfLiteGpuDelegatePluginCApi();
TfLiteOpaqueDelegate* gpu_delegate = pluginCApi->create(tflite_settings);
```
## Quantized models {:#quantized-models}
Android GPU delegate libraries support quantized models by default. You do not
have to make any code changes to use quantized models with the GPU delegate. The
following section explains how to disable quantized support for testing or
experimental purposes.
#### Disable quantized model support
The following code shows how to ***disable*** support for quantized models.
<div>
<devsite-selector>
<section>
<h3>C++</h3>
<p><pre class="prettyprint lang-c++">
TfLiteGpuDelegateOptionsV2 options = TfLiteGpuDelegateOptionsV2Default();
options.experimental_flags = TFLITE_GPU_EXPERIMENTAL_FLAGS_NONE;
auto* delegate = TfLiteGpuDelegateV2Create(&options);
builder.AddDelegate(delegate);
</pre></p>
</section>
</devsite-selector>
</div>
For more information about running quantized models with GPU acceleration, see
[GPU delegate](https://ai.google.dev/edge/litert/performance/gpu#quantized_models)
overview.
tensorflow/lite/g3doc/android/delegates/gpu_task.md
+164
View File
@@ -0,0 +1,164 @@
# GPU acceleration delegate with Task library
Using graphics processing units (GPUs) to run your machine learning (ML) models
can dramatically improve the performance and the user experience
of your ML-enabled applications. On Android devices, you can enable
GPU-accelerated execution of your models using a
[*delegate*](../../performance/delegates) and one of the following APIs:
- Interpreter API - [guide](./gpu)
- Task library API - this guide
- Native (C/C++) API - this [guide](./gpu_native)
This page describes how to enable GPU acceleration for TensorFlow Lite models in
Android apps using the Task library.
For more information about the GPU delegate for TensorFlow Lite,
including best practices and advanced techniques, see the
[GPU delegates](../../performance/gpu) page.
## Use GPU with TensorFlow Lite with Google Play services
The TensorFlow Lite
[Task Libraries](../../inference_with_metadata/task_library/overview) provide a
set of task-specific APIs for building machine learning applications. This
section describes how to use the GPU accelerator delegate with these APIs using
TensorFlow Lite with Google Play services.
[TensorFlow Lite with Google Play services](../play_services) is the recommended
path to use TensorFlow Lite on Android. If your application is targeting devices
not running Google Play, see the
[GPU with Task Library and standalone TensorFlow Lite](#standalone)
section.
### Add project dependencies
To enable access to the GPU delegate with the TensorFlow Lite Task
Libraries using Google Play services, add
`com.google.android.gms:play-services-tflite-gpu` to the
dependencies of your app's `build.gradle` file:
```
dependencies {
...
implementation 'com.google.android.gms:play-services-tflite-gpu:16.4.0'
}
```
### Enable GPU acceleration
Then, verify asynchronously that GPU delegate is available for the device using
the
[`TfLiteGpu`](https://developers.google.com/android/reference/com/google/android/gms/tflite/gpu/support/TfLiteGpu)
class and enable the GPU delegate option for your Task API model class with the
[`BaseOptions`](https://www.tensorflow.org/lite/api_docs/java/org/tensorflow/lite/task/core/BaseOptions.Builder)
class. For example, you can set up GPU in `ObjectDetector` as shown in the
following code examples:
<div>
<devsite-selector>
<section>
<h3>Kotlin</h3>
<p><pre class="prettyprint lang-kotlin">
val useGpuTask = TfLiteGpu.isGpuDelegateAvailable(context)
lateinit val optionsTask = useGpuTask.continueWith { task ->
val baseOptionsBuilder = BaseOptions.builder()
if (task.result) {
baseOptionsBuilder.useGpu()
}
ObjectDetectorOptions.builder()
.setBaseOptions(baseOptionsBuilder.build())
.setMaxResults(1)
.build()
}
</pre></p>
</section>
<section>
<h3>Java</h3>
<p><pre class="prettyprint lang-java">
Task<Boolean> useGpuTask = TfLiteGpu.isGpuDelegateAvailable(context);
Task<ObjectDetectorOptions> optionsTask = useGpuTask.continueWith({ task ->
BaseOptions baseOptionsBuilder = BaseOptions.builder();
if (task.getResult()) {
baseOptionsBuilder.useGpu();
}
return ObjectDetectorOptions.builder()
.setBaseOptions(baseOptionsBuilder.build())
.setMaxResults(1)
.build()
});
</pre></p>
</section>
</devsite-selector>
</div>
## Use GPU with standalone TensorFlow Lite {:#standalone}
If your application is targets devices which are not running Google Play,
it is possible to bundle the GPU delegate to your application and use it
with the standalone version of TensorFlow Lite.
### Add project dependencies
To enable access to the GPU delegate with the TensorFlow Lite Task
Libraries using the standalone version of TensorFlow Lite, add
`org.tensorflow:tensorflow-lite-gpu-delegate-plugin` to the
dependencies of your app's `build.gradle` file:
```
dependencies {
...
implementation 'org.tensorflow:tensorflow-lite'
implementation 'org.tensorflow:tensorflow-lite-gpu-delegate-plugin'
}
```
### Enable GPU acceleration
Then enable the GPU delegate option for your Task API model class with the
[`BaseOptions`](https://www.tensorflow.org/lite/api_docs/java/org/tensorflow/lite/task/core/BaseOptions.Builder)
class. For example, you can set up GPU in `ObjectDetector` as shown in the
following code examples:
<div>
<devsite-selector>
<section>
<h3>Kotlin</h3>
<p><pre class="prettyprint lang-kotlin">
import org.tensorflow.lite.task.core.BaseOptions
import org.tensorflow.lite.task.gms.vision.detector.ObjectDetector
val baseOptions = BaseOptions.builder().useGpu().build()
val options =
ObjectDetector.ObjectDetectorOptions.builder()
.setBaseOptions(baseOptions)
.setMaxResults(1)
.build()
val objectDetector = ObjectDetector.createFromFileAndOptions(
context, model, options)
</pre></p>
</section>
<section>
<h3>Java</h3>
<p><pre class="prettyprint lang-java">
import org.tensorflow.lite.task.core.BaseOptions
import org.tensorflow.lite.task.gms.vision.detector.ObjectDetector
BaseOptions baseOptions = BaseOptions.builder().useGpu().build();
ObjectDetectorOptions options =
ObjectDetectorOptions.builder()
.setBaseOptions(baseOptions)
.setMaxResults(1)
.build();
val objectDetector = ObjectDetector.createFromFileAndOptions(
context, model, options);
</pre></p>
</section>
</devsite-selector>
</div>
tensorflow/lite/g3doc/android/delegates/hexagon.md
+312
View File
@@ -0,0 +1,312 @@
# TensorFlow Lite Hexagon delegate
<aside class="warning">
<p><b>Warning:</b> The
<a href="https://www.tensorflow.org/lite/android/delegates/nnapi">
NNAPI</a> and <a href="https://www.tensorflow.org/lite/android/delegates/hexagon">
Hexagon</a> delegates are deprecated and no longer supported by TensorFlow
Lite. For more information, see the
<a href="https://developer.android.com/ndk/guides/neuralnetworks/migration-guide">
NNAPI Migration Guide</a> and
<a href="https://www.tensorflow.org/lite/performance/delegates">TF Lite
delegates documentation</a>.</p>
</aside>
This document explains how to use the TensorFlow Lite Hexagon Delegate in your
application using the Java and/or C API. The delegate leverages the Qualcomm
Hexagon library to execute quantized kernels on the DSP. Note that the delegate
is intended to *complement* NNAPI functionality, particularly for devices where
NNAPI DSP acceleration is unavailable (e.g., on older devices, or devices that
don’t yet have a DSP NNAPI driver).
**Supported devices:**
Currently the following Hexagon architecture are supported, including but not
limited to:
* Hexagon 680
* SoC examples: Snapdragon 821, 820, 660
* Hexagon 682
* SoC examples: Snapdragon 835
* Hexagon 685
* SoC examples: Snapdragon 845, Snapdragon 710, QCS410, QCS610, QCS605,
QCS603
* Hexagon 690
* SoC examples: Snapdragon 855, RB5
**Supported models:**
The Hexagon delegate supports all models that conform to our
[8-bit symmetric quantization spec](https://www.tensorflow.org/lite/performance/quantization_spec),
including those generated using
[post-training integer quantization](https://www.tensorflow.org/lite/performance/post_training_integer_quant).
UInt8 models trained with the legacy
[quantization-aware training](https://github.com/tensorflow/tensorflow/tree/r1.13/tensorflow/contrib/quantize)
path are also supported, for example,
[these quantized versions](https://www.tensorflow.org/lite/guide/hosted_models#quantized_models)
on our Hosted Models page.
## Hexagon delegate Java API
```java
public class HexagonDelegate implements Delegate, Closeable {
/*
* Creates a new HexagonDelegate object given the current 'context'.
* Throws UnsupportedOperationException if Hexagon DSP delegation is not
* available on this device.
*/
public HexagonDelegate(Context context) throws UnsupportedOperationException
/**
* Frees TFLite resources in C runtime.
*
* User is expected to call this method explicitly.
*/
@Override
public void close();
}
```
### Example usage
#### Step 1. Edit app/build.gradle to use the nightly Hexagon delegate AAR
```
dependencies {
...
implementation 'org.tensorflow:tensorflow-lite:0.0.0-nightly-SNAPSHOT'
implementation 'org.tensorflow:tensorflow-lite-hexagon:0.0.0-nightly-SNAPSHOT'
}
```
#### Step 2. Add Hexagon libraries to your Android app {:#hexagon_versions}
* Download and run hexagon_nn_skel.run. It should provide 3 different shared
libraries “libhexagon_nn_skel.so”, “libhexagon_nn_skel_v65.so”,
“libhexagon_nn_skel_v66.so”
* [v1.10.3](https://storage.cloud.google.com/download.tensorflow.org/tflite/hexagon_nn_skel_1_10_3_1.run)
* [v1.14](https://storage.cloud.google.com/download.tensorflow.org/tflite/hexagon_nn_skel_v1.14.run)
* [v1.17](https://storage.cloud.google.com/download.tensorflow.org/tflite/hexagon_nn_skel_v1.17.0.0.run)
* [v1.20](https://storage.cloud.google.com/download.tensorflow.org/tflite/hexagon_nn_skel_v1.20.0.0.run)
* [v1.20.0.1](https://storage.cloud.google.com/download.tensorflow.org/tflite/hexagon_nn_skel_v1.20.0.1.run)
Caution: The currently released versions of the Hexagon delegate, up to version
1.20.0.1, are no longer supported. An updated version of this delegate is
expected soon.
Note: You must accept the license agreement before using the delegate.
Note: You must use the hexagon_nn libraries with the compatible version of
interface library. Interface library is part of the AAR and fetched by bazel
through the
[config](https://github.com/tensorflow/tensorflow/blob/master/third_party/hexagon/workspace.bzl)
The version in the bazel config is the version you should use.
* Include all 3 in your app with other shared libraries. See
[How to add shared library to your app](#how-to-add-shared-library-to-your-app).
The delegate will automatically pick the one with best performance depending
on the device.
Note: If your app will be built for both 32 and 64-bit ARM devices, then
add the Hexagon shared libs to both 32 and 64-bit lib folders.
#### Step 3. Create a delegate and initialize a TensorFlow Lite Interpreter
```java
import org.tensorflow.lite.HexagonDelegate;
// Create the Delegate instance.
try {
hexagonDelegate = new HexagonDelegate(activity);
tfliteOptions.addDelegate(hexagonDelegate);
} catch (UnsupportedOperationException e) {
// Hexagon delegate is not supported on this device.
}
tfliteInterpreter = new Interpreter(tfliteModel, tfliteOptions);
// Dispose after finished with inference.
tfliteInterpreter.close();
if (hexagonDelegate != null) {
hexagonDelegate.close();
}
```
## Hexagon delegate C API
```c
struct TfLiteHexagonDelegateOptions {
// This corresponds to the debug level in the Hexagon SDK. 0 (default)
// means no debug.
int debug_level;
// This corresponds to powersave_level in the Hexagon SDK.
// where 0 (default) means high performance which means more power
// consumption.
int powersave_level;
// If set to true, performance information about the graph will be dumped
// to Standard output, this includes cpu cycles.
// WARNING: Experimental and subject to change anytime.
bool print_graph_profile;
// If set to true, graph structure will be dumped to Standard output.
// This is usually beneficial to see what actual nodes executed on
// the DSP. Combining with 'debug_level' more information will be printed.
// WARNING: Experimental and subject to change anytime.
bool print_graph_debug;
};
// Return a delegate that uses Hexagon SDK for ops execution.
// Must outlive the interpreter.
TfLiteDelegate*
TfLiteHexagonDelegateCreate(const TfLiteHexagonDelegateOptions* options);
// Do any needed cleanup and delete 'delegate'.
void TfLiteHexagonDelegateDelete(TfLiteDelegate* delegate);
// Initializes the DSP connection.
// This should be called before doing any usage of the delegate.
// "lib_directory_path": Path to the directory which holds the
// shared libraries for the Hexagon NN libraries on the device.
void TfLiteHexagonInitWithPath(const char* lib_directory_path);
// Same as above method but doesn't accept the path params.
// Assumes the environment setup is already done. Only initialize Hexagon.
Void TfLiteHexagonInit();
// Clean up and switch off the DSP connection.
// This should be called after all processing is done and delegate is deleted.
Void TfLiteHexagonTearDown();
```
### Example usage
#### Step 1. Edit app/build.gradle to use the nightly Hexagon delegate AAR
```
dependencies {
...
implementation 'org.tensorflow:tensorflow-lite:0.0.0-nightly-SNAPSHOT'
implementation 'org.tensorflow:tensorflow-lite-hexagon:0.0.0-nightly-SNAPSHOT'
}
```
#### Step 2. Add Hexagon libraries to your Android app
* Download and run hexagon_nn_skel.run. It should provide 3 different shared
libraries “libhexagon_nn_skel.so”, “libhexagon_nn_skel_v65.so”,
“libhexagon_nn_skel_v66.so”
* [v1.10.3](https://storage.cloud.google.com/download.tensorflow.org/tflite/hexagon_nn_skel_1_10_3_1.run)
* [v1.14](https://storage.cloud.google.com/download.tensorflow.org/tflite/hexagon_nn_skel_v1.14.run)
* [v1.17](https://storage.cloud.google.com/download.tensorflow.org/tflite/hexagon_nn_skel_v1.17.0.0.run)
* [v1.20](https://storage.cloud.google.com/download.tensorflow.org/tflite/hexagon_nn_skel_v1.20.0.0.run)
* [v1.20.0.1](https://storage.cloud.google.com/download.tensorflow.org/tflite/hexagon_nn_skel_v1.20.0.1.run)
Caution: The currently released versions of the Hexagon delegate, up to version
1.20.0.1, are no longer supported. An updated version of this delegate is
expected soon.
Note: You must accept the license agreement before using the delegate.
Note: You must use the hexagon_nn libraries with the compatible version of
interface library. Interface library is part of the AAR and fetched by bazel
through the
[config](https://github.com/tensorflow/tensorflow/blob/master/third_party/hexagon/workspace.bzl).
The version in the bazel config is the version you should use.
* Include all 3 in your app with other shared libraries. See
[How to add shared library to your app](#how-to-add-shared-library-to-your-app).
The delegate will automatically pick the one with best performance depending
on the device.
Note: If your app will be built for both 32 and 64-bit ARM devices, then
add the Hexagon shared libs to both 32 and 64-bit lib folders.
#### Step 3. Include the C header
* The header file "hexagon_delegate.h" can be downloaded from
[GitHub](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/delegates/hexagon/hexagon_delegate.h)
or extracted from the Hexagon delegate AAR.
#### Step 4. Create a delegate and initialize a TensorFlow Lite Interpreter
* In your code, ensure the native Hexagon library is loaded. This can be done
by calling `System.loadLibrary("tensorflowlite_hexagon_jni");` \
in your Activity or Java entry-point.
* Create a delegate, example:
```c
#include "tensorflow/lite/delegates/hexagon/hexagon_delegate.h"
// Assuming shared libraries are under "/data/local/tmp/"
// If files are packaged with native lib in android App then it
// will typically be equivalent to the path provided by
// "getContext().getApplicationInfo().nativeLibraryDir"
const char[] library_directory_path = "/data/local/tmp/";
TfLiteHexagonInitWithPath(library_directory_path); // Needed once at startup.
::tflite::TfLiteHexagonDelegateOptions params = {0};
// 'delegate_ptr' Need to outlive the interpreter. For example,
// If your use case requires resizing the input or anything that can trigger
// re-applying delegates then 'delegate_ptr' must outlive the interpreter.
auto* delegate_ptr = ::tflite::TfLiteHexagonDelegateCreate(&params);
Interpreter::TfLiteDelegatePtr delegate(delegate_ptr,
[](TfLiteDelegate* delegate) {
::tflite::TfLiteHexagonDelegateDelete(delegate);
});
interpreter->ModifyGraphWithDelegate(delegate.get());
// After usage of delegate.
TfLiteHexagonTearDown(); // Needed once at end of app/DSP usage.
```
## Add the shared library to your app
* Create folder “app/src/main/jniLibs”, and create a directory for each target
architecture. For example,
* ARM 64-bit: `app/src/main/jniLibs/arm64-v8a`
* ARM 32-bit: `app/src/main/jniLibs/armeabi-v7a`
* Put your .so in the directory that match the architecture.
Note: If you're using App Bundle for publishing your Application, you might want
to set android.bundle.enableUncompressedNativeLibs=false in the
gradle.properties file.
## Feedback
For issues, please create a
[GitHub](https://github.com/tensorflow/tensorflow/issues/new?template=50-other-issues.md)
issue with all the necessary repro details, including the phone model and board
used (`adb shell getprop ro.product.device` and `adb shell getprop
ro.board.platform`).
## FAQ
* Which ops are supported by the delegate?
* See the current list of
[supported ops and constraints](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/delegates/hexagon/README.md)
* How can I tell that the model is using the DSP when I enable the delegate?
* Two log messages will be printed when you enable the delegate - one to
indicate if the delegate was created and another to indicate how many
nodes are running using the delegate. \
`Created TensorFlow Lite delegate for Hexagon.` \
`Hexagon delegate: X nodes delegated out of Y nodes.`
* Do I need all Ops in the model to be supported to run the delegate?
* No, the Model will be partitioned into subgraphs based on the supported
ops. Any unsupported ops will run on the CPU.
* How can I build the Hexagon delegate AAR from source?
* Use `bazel build -c opt --config=android_arm64
tensorflow/lite/delegates/hexagon/java:tensorflow-lite-hexagon`.
* Why does Hexagon delegate fail to initialize although my Android device has
a supported SoC?
* Verify if your device indeed has a supported SoC. Run `adb shell cat
/proc/cpuinfo | grep Hardware` and see if it returns something like
"Hardware : Qualcomm Technologies, Inc MSMXXXX".
* Some phone manufacturers use different SoCs for the same phone model.
Therefore, Hexagon delegate may only work on some but not all devices of
the same phone model.
* Some phone manufactures intentionally restrict the use of Hexagon DSP
from non-system Android apps, making the Hexagon delegate unable to
work.
* My phone has locked DSP access. I rooted the phone and still can't run the
delegate, what to do ?
* Make sure to disable SELinux enforce by running `adb shell setenforce 0`
tensorflow/lite/g3doc/android/delegates/nnapi.md
+264
View File
@@ -0,0 +1,264 @@
# TensorFlow Lite NNAPI delegate
<aside class="warning">
<p><b>Warning:</b> The
<a href="https://www.tensorflow.org/lite/android/delegates/nnapi">
NNAPI</a> and <a href="https://www.tensorflow.org/lite/android/delegates/hexagon">
Hexagon</a> delegates are deprecated and no longer supported by TensorFlow
Lite. For more information, see the
<a href="https://developer.android.com/ndk/guides/neuralnetworks/migration-guide">
NNAPI Migration Guide</a> and
<a href="https://www.tensorflow.org/lite/performance/delegates">TF Lite
delegates documentation</a>.</p>
</aside>
The
[Android Neural Networks API (NNAPI)](https://developer.android.com/ndk/guides/neuralnetworks)
is available on all Android devices running Android 8.1 (API level 27) or
higher. It provides acceleration for TensorFlow Lite models on Android devices
with supported hardware accelerators including:
* Graphics Processing Unit (GPU)
* Digital Signal Processor (DSP)
* Neural Processing Unit (NPU)
Performance will vary depending on the specific hardware available on device.
This page describes how to use the NNAPI delegate with the TensorFlow Lite
Interpreter in Java and Kotlin. For Android C APIs, please refer to
[Android Native Developer Kit documentation](https://developer.android.com/ndk/guides/neuralnetworks).
## Trying the NNAPI delegate on your own model
### Gradle import
The NNAPI delegate is part of the TensorFlow Lite Android interpreter, release
1.14.0 or higher. You can import it to your project by adding the following to
your module gradle file:
```groovy
dependencies {
implementation 'org.tensorflow:tensorflow-lite:+'
}
```
### Initializing the NNAPI delegate
Add the code to initialize the NNAPI delegate before you initialize the
TensorFlow Lite interpreter.
Note: Although NNAPI is supported from API Level 27 (Android Oreo MR1), the
support for operations improved significantly for API Level 28 (Android Pie)
onwards. As a result, we recommend developers use the NNAPI delegate for Android
Pie or above for most scenarios.
#### kotlin
```kotlin
import android.content.res.AssetManager
import org.tensorflow.lite.Interpreter
import org.tensorflow.lite.nnapi.NnApiDelegate
import java.io.FileInputStream
import java.io.IOException
import java.nio.MappedByteBuffer
import java.nio.channels.FileChannel
...
val options = Interpreter.Options()
var nnApiDelegate: NnApiDelegate? = null
// Initialize interpreter with NNAPI delegate for Android Pie or above
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.P) {
nnApiDelegate = NnApiDelegate()
options.addDelegate(nnApiDelegate)
}
val assetManager = assets
// Initialize TFLite interpreter
val tfLite: Interpreter
try {
tfLite = Interpreter(loadModelFile(assetManager, "model.tflite"), options)
} catch (e: Exception) {
throw RuntimeException(e)
}
// Run inference
// ...
// Unload delegate
tfLite.close()
nnApiDelegate?.close()
...
@Throws(IOException::class)
private fun loadModelFile(assetManager: AssetManager, modelFilename: String): MappedByteBuffer {
val fileDescriptor = assetManager.openFd(modelFilename)
val inputStream = FileInputStream(fileDescriptor.fileDescriptor)
val fileChannel = inputStream.channel
val startOffset = fileDescriptor.startOffset
val declaredLength = fileDescriptor.declaredLength
return fileChannel.map(FileChannel.MapMode.READ_ONLY, startOffset, declaredLength)
}
...
```
#### java
```java
import android.content.res.AssetManager;
import org.tensorflow.lite.Interpreter;
import org.tensorflow.lite.nnapi.NnApiDelegate;
import java.io.FileInputStream;
import java.io.IOException;
import java.nio.MappedByteBuffer;
import java.nio.channels.FileChannel;
...
Interpreter.Options options = (new Interpreter.Options());
NnApiDelegate nnApiDelegate = null;
// Initialize interpreter with NNAPI delegate for Android Pie or above
if(Build.VERSION.SDK_INT >= Build.VERSION_CODES.P) {
nnApiDelegate = new NnApiDelegate();
options.addDelegate(nnApiDelegate);
}
AssetManager assetManager = getAssets();
// Initialize TFLite interpreter
try {
tfLite = new Interpreter(loadModelFile(assetManager, "model.tflite"), options);
} catch (Exception e) {
throw new RuntimeException(e);
}
// Run inference
// ...
// Unload delegate
tfLite.close();
if(null != nnApiDelegate) {
nnApiDelegate.close();
}
...
private MappedByteBuffer loadModelFile(AssetManager assetManager, String modelFilename) throws IOException {
AssetFileDescriptor fileDescriptor = assetManager.openFd(modelFilename);
FileInputStream inputStream = new FileInputStream(fileDescriptor.getFileDescriptor());
FileChannel fileChannel = inputStream.getChannel();
long startOffset = fileDescriptor.getStartOffset();
long declaredLength = fileDescriptor.getDeclaredLength();
return fileChannel.map(FileChannel.MapMode.READ_ONLY, startOffset, declaredLength);
}
...
```
## Best practices
### Test performance before deploying
Runtime performance can vary significantly due to model architecture, size,
operations, hardware availability, and runtime hardware utilization. For
example, if an app heavily utilizes the GPU for rendering, NNAPI acceleration
may not improve performance due to resource contention. We recommend running a
simple performance test using the debug logger to measure inference time. Run
the test on several phones with different chipsets (manufacturer or models from
the same manufacturer) that are representative of your user base before enabling
NNAPI in production.
For advanced developers, TensorFlow Lite also offers
[a model benchmark tool for Android](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/tools/benchmark).
### Create a device exclusion list
In production, there may be cases where NNAPI does not perform as expected. We
recommend developers maintain a list of devices that should not use NNAPI
acceleration in combination with particular models. You can create this list
based on the value of `"ro.board.platform"`, which you can retrieve using the
following code snippet:
```java
String boardPlatform = "";
try {
Process sysProcess =
new ProcessBuilder("/system/bin/getprop", "ro.board.platform").
redirectErrorStream(true).start();
BufferedReader reader = new BufferedReader
(new InputStreamReader(sysProcess.getInputStream()));
String currentLine = null;
while ((currentLine=reader.readLine()) != null){
boardPlatform = line;
}
sysProcess.destroy();
} catch (IOException e) {}
Log.d("Board Platform", boardPlatform);
```
For advanced developers, consider maintaining this list via a remote
configuration system. The TensorFlow team is actively working on ways to
simplify and automate discovering and applying the optimal NNAPI configuration.
### Quantization
Quantization reduces model size by using 8-bit integers or 16-bit floats instead
of 32-bit floats for computation. 8-bit integer model sizes are a quarter of the
32-bit float versions; 16-bit floats are half of the size. Quantization can
improve performance significantly though the process could trade off some model
accuracy.
There are multiple types of post-training quantization techniques available,
but, for maximum support and acceleration on current hardware, we recommend
[full integer quantization](post_training_quantization#full_integer_quantization_of_weights_and_activations).
This approach converts both the weight and the operations into integers. This
quantization process requires a representative dataset to work.
### Use supported models and ops
If the NNAPI delegate does not support some of the ops or parameter combinations
in a model, the framework only runs the supported parts of the graph on the
accelerator. The remainder runs on the CPU, which results in split execution.
Due to the high cost of CPU/accelerator synchronization, this may result in
slower performance than executing the whole network on the CPU alone.
NNAPI performs best when models only use
[supported ops](https://developer.android.com/ndk/guides/neuralnetworks#model).
The following models are known to be compatible with NNAPI:
* [MobileNet v1 (224x224) image classification (float model download)](https://ai.googleblog.com/2017/06/mobilenets-open-source-models-for.html)
[(quantized model download)](http://download.tensorflow.org/models/mobilenet_v1_2018_08_02/mobilenet_v1_1.0_224_quant.tgz)
\
_(image classification model designed for mobile and embedded based vision
applications)_
* [MobileNet v2 SSD object detection](https://ai.googleblog.com/2018/07/accelerated-training-and-inference-with.html)
[(download)](https://storage.googleapis.com/download.tensorflow.org/models/tflite/gpu/mobile_ssd_v2_float_coco.tflite)
\
_(image classification model that detects multiple objects with bounding
boxes)_
* [MobileNet v1(300x300) Single Shot Detector (SSD) object detection](https://ai.googleblog.com/2018/07/accelerated-training-and-inference-with.html)
[(download)] (https://storage.googleapis.com/download.tensorflow.org/models/tflite/coco_ssd_mobilenet_v1_1.0_quant_2018_06_29.zip)
* [PoseNet for pose estimation](https://github.com/tensorflow/tfjs-models/tree/master/posenet)
[(download)](https://storage.googleapis.com/download.tensorflow.org/models/tflite/gpu/multi_person_mobilenet_v1_075_float.tflite)
\
_(vision model that estimates the poses of a person(s) in image or video)_
NNAPI acceleration is also not supported when the model contains
dynamically-sized outputs. In this case, you will get a warning like:
```none
ERROR: Attempting to use a delegate that only supports static-sized tensors \
with a graph that has dynamic-sized tensors.
```
### Enable NNAPI CPU implementation
A graph that can't be processed completely by an accelerator can fall back to
the NNAPI CPU implementation. However, since this is typically less performant
than the TensorFlow interpreter, this option is disabled by default in the NNAPI
delegate for Android 10 (API Level 29) or above. To override this behavior, set
`setUseNnapiCpu` to `true` in the `NnApiDelegate.Options` object.