wehub/tensorflow--tensorflow
1
0
Fork 0
Code Issues Pull Requests Actions 6 Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution
cffconvert / validate (push) Has been skipped
Details
License Check / license-check (push) Failing after 2s
Details

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 12:14:16 +08:00
commit 8a852e4b4e
36502 changed files with 9277225 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tensorflow/lite/g3doc/android/acceleration_service.md
+339
View File
@@ -0,0 +1,339 @@
# Acceleration Service for Android (Beta)
Beta: Acceleration Service for Android is currently in Beta.
Please review the [Caveats](#caveats) and the [Terms and Privacy]
(#terms_privacy) sections of this page for more details.
The use of specialized processors such as GPUs, NPUs or DSPs for hardware
acceleration can dramatically improve inference performance (up to 10x faster
inference in some cases) and the user experience of your ML-enabled Android
application. However, given the variety of hardware and drivers your users might
have, picking the optimal hardware acceleration configuration for each user's
device can be challenging. Moreover, enabling the wrong configuration on a
device can create poor user experience due to high latency or, in some rare
cases, runtime errors or accuracy issues caused by hardware incompatibilities.
Acceleration Service for Android is an API that helps you pick the
optimal hardware acceleration configuration for a given user device and your
`.tflite` model, while minimizing the risk of runtime error or accuracy issues.
Acceleration Service evaluates different acceleration configurations on user
devices by running internal inference benchmarks with your TensorFlow Lite
model. These test runs typically complete in a few seconds, depending on your
model. You can run the benchmarks once on every user device before inference
time, cache the result and use it during inference. These benchmarks are run
out-of-process; which minimizes the risk of crashes to your app.
Provide your model, data samples and expected results ("golden" inputs and
outputs) and Acceleration Service will run an internal TFLite inference
benchmark to provide you with hardware recommendations.
![image](../images/acceleration/acceleration_service.png)
Acceleration Service is part of Android's custom ML stack and works with
[TensorFlow Lite in Google Play services](https://www.tensorflow.org/lite/android/play_services).
## Add the dependencies to your project
Add the following dependencies to your application's build.gradle file:
```
implementation "com.google.android.gms:play-services-tflite-
acceleration-service:16.0.0-beta01"
```
The Acceleration Service API works with [TensorFlow Lite in Google Play
Services](https://www.tensorflow.org/lite/android/play_services). If you
aren't using the TensorFlow Lite runtime provided via Play Services yet, you
will need to update your [dependencies](https://www.tensorflow.org/lite/android/play_services#1_add_project_dependencies_2).
## How to use the Acceleration Service API
To use Acceleration Service, start by creating the acceleration configuration
you want to evaluate for you model (e.g GPU with OpenGL). Then create a
validation configuration with your model, some sample data and the expected
model output. Finally call `validateConfig()` in passing both your
acceleration configuration and validation configuration.
![image](../images/acceleration/acceleration_service_steps.png)
### Create acceleration configurations
Acceleration configurations are representations of the hardware configurations
which are translated into delegates during the execution time.
The Acceleration Service will then use these configurations internally
to perform test inferences.
At the moment the acceleration service enables you to evaluate GPU
configurations (converted to GPU delegate during the execution time)
with the
[GpuAccelerationConfig](https://developers.google.com/android/reference/com/google/android/gms/tflite/acceleration/GpuAccelerationConfig)
and CPU inference (with
[CpuAccelerationConfig](https://developers.google.com/android/reference/com/google/android/gms/tflite/acceleration/CpuAccelerationConfig)).
We are working on supporting more delegates to access other hardware in the
future.
#### GPU acceleration configuration
Create a GPU acceleration configuration as follow:
```
AccelerationConfig accelerationConfig = new GpuAccelerationConfig.Builder()
.setEnableQuantizedInference(false)
.build();
```
You must specify whether or not your model is using quantization with
[`setEnableQuantizedInference()`](https://developers.google.com/android/reference/com/google/android/gms/tflite/acceleration/GpuAccelerationConfig.Builder#public-gpuaccelerationconfig.builder-setenablequantizedinference-boolean-value).
#### CPU acceleration configuration
Create the CPU acceleration as follow:
```
AccelerationConfig accelerationConfig = new CpuAccelerationConfig.Builder()
.setNumThreads(2)
.build();
```
Use the
[`setNumThreads()`](https://developers.google.com/android/reference/com/google/android/gms/tflite/acceleration/CpuAccelerationConfig.Builder#setNumThreads\(int\))
method to define the number of threads you want to use to evaluate CPU
inference.
### Create validation configurations
Validation configurations enable you to define how you want the Acceleration
Service to evaluate inferences. You will use them to pass:
- input samples,
- expected outputs,
- accuracy validation logic.
Make sure to provide input samples for which you expect a good performance of
your model (also known as “golden” samples).
Create a
[`ValidationConfig`](https://developers.google.com/android/reference/com/google/android/gms/tflite/acceleration/ValidationConfig)
with
[`CustomValidationConfig.Builder`](https://developers.google.com/android/reference/com/google/android/gms/tflite/acceleration/CustomValidationConfig.Builder)
as follow:
```
ValidationConfig validationConfig = new CustomValidationConfig.Builder()
.setBatchSize(5)
.setGoldenInputs(inputs)
.setGoldenOutputs(outputBuffer)
.setAccuracyValidator(new MyCustomAccuracyValidator())
.build();
```
Specify the number of the golden samples with
[`setBatchSize()`](https://developers.google.com/android/reference/com/google/android/gms/tflite/acceleration/CustomValidationConfig.Builder#setBatchSize\(int\)).
Pass the inputs of your golden samples using
[`setGoldenInputs()`](https://developers.google.com/android/reference/com/google/android/gms/tflite/acceleration/CustomValidationConfig.Builder#public-customvalidationconfig.builder-setgoldeninputs-object...-value).
Provide the expected output for the input passed with
[`setGoldenOutputs()`](https://developers.google.com/android/reference/com/google/android/gms/tflite/acceleration/CustomValidationConfig.Builder#public-customvalidationconfig.builder-setgoldenoutputs-bytebuffer...-value).
You can define a maximum inference time with [`setInferenceTimeoutMillis()`](
https://developers.google.com/android/reference/com/google/android/gms/tflite/acceleration/CustomValidationConfig.Builder#public-customvalidationconfig.builder-setinferencetimeoutmillis-long-value)
(5000 ms by default). If the inference takes longer than the time you defined,
the configuration will be rejected.
Optionally, you can also create a custom [`AccuracyValidator`](https://developers.google.com/android/reference/com/google/android/gms/tflite/acceleration/CustomValidationConfig.AccuracyValidator)
as follow:
```
class MyCustomAccuracyValidator implements AccuracyValidator {
boolean validate(
BenchmarkResult benchmarkResult,
ByteBuffer[] goldenOutput) {
for (int i = 0; i < benchmarkResult.actualOutput().size(); i++) {
if (!goldenOutputs[i]
.equals(benchmarkResult.actualOutput().get(i).getValue())) {
return false;
}
}
return true;
}
}
```
Make sure to define a validation logic that works for your use case.
Note that if the validation data is already embedded in your model, you can use
[`EmbeddedValidationConfig`](https://developers.google.com/android/reference/com/google/android/gms/tflite/acceleration/EmbeddedValidationConfig).
##### Generate validation outputs
Golden outputs are optional and as long as you provide golden inputs, the
Acceleration Service can internally generate the golden outputs. You can also
define the acceleration configuration used to generate these golden outputs by
calling [`setGoldenConfig()`](https://developers.google.com/android/reference/com/google/android/gms/tflite/acceleration/CustomValidationConfig.Builder#setGoldenConfig(com.google.android.gms.tflite.acceleration.AccelerationConfig)):
```
ValidationConfig validationConfig = new CustomValidationConfig.Builder()
.setBatchSize(5)
.setGoldenInputs(inputs)
.setGoldenConfig(customCpuAccelerationConfig)
[...]
.build();
```
### Validate Acceleration configuration
Once you have created an acceleration configuration and a validation config you
can evaluate them for your model.
Make sure that the TensorFlow Lite with Play Services runtime is properly
initialized and that the GPU delegate is available for the device by running:
```
TfLiteGpu.isGpuDelegateAvailable(context)
.onSuccessTask(gpuAvailable -> TfLite.initialize(context,
TfLiteInitializationOptions.builder()
.setEnableGpuDelegateSupport(gpuAvailable)
.build()
)
);
```
Instantiate the [`AccelerationService`](https://developers.google.com/android/reference/com/google/android/gms/tflite/acceleration/AccelerationService)
by calling [`AccelerationService.create()`](https://developers.google.com/android/reference/com/google/android/gms/tflite/acceleration/AccelerationService#create(android.content.Context)).
You can then validate your acceleration configuration for your model by calling
[`validateConfig()`](https://developers.google.com/android/reference/com/google/android/gms/tflite/acceleration/AccelerationService#validateConfig(com.google.android.gms.tflite.acceleration.Model,%20com.google.android.gms.tflite.acceleration.AccelerationConfig,%20com.google.android.gms.tflite.acceleration.ValidationConfig)):
```
InterpreterApi interpreter;
InterpreterOptions interpreterOptions = InterpreterApi.Options();
AccelerationService.create(context)
.validateConfig(model, accelerationConfig, validationConfig)
.addOnSuccessListener(validatedConfig -> {
if (validatedConfig.isValid() && validatedConfig.benchmarkResult().hasPassedAccuracyTest()) {
interpreterOptions.setAccelerationConfig(validatedConfig);
interpreter = InterpreterApi.create(model, interpreterOptions);
});
```
You can also validate multiple configurations by calling
[`validateConfigs()`](https://developers.google.com/android/reference/com/google/android/gms/tflite/acceleration/AccelerationService#validateConfigs(com.google.android.gms.tflite.acceleration.Model,%20java.lang.Iterable%3Ccom.google.android.gms.tflite.acceleration.AccelerationConfig%3E,%20com.google.android.gms.tflite.acceleration.ValidationConfig))
and passing an `Iterable<AccelerationConfig>` object as a parameter.
`validateConfig()`will return a
`Task<`[`ValidatedAccelerationConfigResult`](https://developers.google.com/android/reference/com/google/android/gms/tflite/acceleration/ValidatedAccelerationConfigResult)`>`
from the Google Play services
[Task Api](https://developers.google.com/android/guides/tasks) which enables
asynchronous tasks. \
To get the result from the validation call, add an
[`addOnSuccessListener()`](https://developers.google.com/android/reference/com/google/android/gms/tasks/OnSuccessListener)
callback.
#### Use validated configuration in your interpreter
After checking if the `ValidatedAccelerationConfigResult` returned in the
callback is valid, you can set the validated config as an acceleration config
for your interpreter calling `interpreterOptions.setAccelerationConfig()`.
#### Configuration caching
The optimal acceleration configuration for your model is unlikely to change on
the device. So once you receive a satisfying acceleration configuration, you
should store it on the device and let your application retrieve it and use it to
create your `InterpreterOptions` during the following sessions instead of
running another validation. The `serialize()` and `deserialize()` methods in
`ValidatedAccelerationConfigResult` make the storage and retrieval process
easier.
### Sample application
To review an in-situ integration of the Acceleration Service, take a look at the
[sample app](https://github.com/tensorflow/examples/tree/master/lite/examples/acceleration_service/android_play_services).
## Limitations
The Acceleration Service has the current following limitations:
- Only CPU and GPU acceleration configurations are supported at the moment,
- It only supports TensorFlow Lite in Google Play services and you cannot
use it if you are using the bundled version of TensorFlow Lite,
- It doesn't support the TensorFlow Lite [Task
Library](https://www.tensorflow.org/lite/inference_with_metadata/task_library/overview)
as you can't directly initialize
[`BaseOptions`](https://www.tensorflow.org/lite/api_docs/java/org/tensorflow/lite/task/core/BaseOptions.Builder)
with the `ValidatedAccelerationConfigResult` object.
- Acceleration Service SDK only supports API level 22 and above.
## Caveats {:#caveats}
Please review the following caveats carefully, especially if you are planning
to use this SDK in production:
- Before exiting Beta and releasing the stable version for the
Acceleration Service API, we will publish a new SDK which may have some
differences from the current Beta one. In order to continue using the
Acceleration Service, you will need to migrate to this new SDK and push an
update to your app in a timely manner. Not doing so may cause breakages as
the Beta SDK may no longer be compatible with Google Play services after
some time.
- There is no guarantee that a specific feature within the Acceleration
Service API or the API as a whole will ever become generally available. It
may remain in Beta indefinitely, be shut down, or be combined with other
features into packages designed for specific developer audiences. Some
features with the Acceleration Service API or the entire API itself may
eventually become generally available, but there is no fixed schedule for
this.
## Terms and privacy {:#terms_privacy}
#### Terms of Service
Use of the Acceleration Service APIs is subject to the [Google APIs Terms of
Service](https://developers.google.com/terms/).\
Additionally, the Acceleration Service APIs is currently in beta
and, as such, by using it you acknowledge the potential issues outlined in the
Caveats section above and acknowledge that the Acceleration Service may not
always perform as specified.
#### Privacy
When you use the Acceleration Service APIs, processing of the input data (e.g.
images, video, text) fully happens on-device, and **the Acceleration Service
does not send that data to Google servers**. As a result, you can use our APIs
for processing input data that should not leave the device.\
The Acceleration Service APIs may contact Google servers from time to time in
order to receive things like bug fixes, updated models and hardware accelerator
compatibility information. The Acceleration Service APIs also send metrics about
the performance and utilization of the APIs in your app to Google. Google uses
this metrics data to measure performance, debug, maintain and improve the APIs,
and detect misuse or abuse, as further described in our [Privacy
Policy](https://policies.google.com/privacy).\
**You are responsible for informing users of your app about Google's processing
of the Acceleration Service metrics data as required by applicable law.**\
Data we collect includes the following:
- Device information (such as manufacturer, model, OS version and build) and
available ML hardware accelerators (GPU and DSP). Used for diagnostics and
usage analytics.
- App information (package name / bundle id, app version). Used for
diagnostics and usage analytics.
- API configuration (such as image format and resolution). Used for
diagnostics and usage analytics.
- Event type (such as initialize, download model, update, run, detection).
Used for diagnostics and usage analytics.
- Error codes. Used for diagnostics.
- Performance metrics. Used for diagnostics.
- Per-installation identifiers that do not uniquely identify a user or
physical device. Used for operation of remote configuration and usage
analytics.
- Network request sender IP addresses. Used for remote configuration
diagnostics. Collected IP addresses are retained temporarily.
## Support and feedback
You can provide feedback and get support through the TensorFlow Issue Tracker.
Please report issues and support requests using the
[issue template](https://github.com/tensorflow/tensorflow/issues/new?title=TensorFlow+Lite+in+Play+Services+issue&template=tflite-in-play-services.md)
for TensorFlow Lite in Google Play services.
tensorflow/lite/g3doc/android/delegates/gpu.md
+241
View File
@@ -0,0 +1,241 @@
# GPU acceleration delegate with Java/Kotlin Interpreter API
Using graphics processing units (GPUs) to run your machine learning (ML) models
can dramatically improve the performance and the user experience
of your ML-enabled applications. On Android devices, you can enable
[*delegate*](../../performance/delegates) and one of the following APIs:
- Java/Kotlin Interpreter API - this guide
- Task library API - [guide](./gpu_task)
- Native (C/C++) API - [guide](./gpu_native)
This page describes how to enable GPU acceleration for TensorFlow Lite models in
Android apps using the Interpreter API.
For more information about using the GPU delegate for
TensorFlow Lite, including best practices and advanced techniques, see the
[GPU delegates](../../performance/gpu) page.
## Use GPU with TensorFlow Lite with Google Play services
The TensorFlow Lite Java/Kotlin [Interpreter API](https://tensorflow.org/lite/api_docs/java/org/tensorflow/lite/InterpreterApi)
provides a set of general purpose APIs for building a machine learning
applications. This section describes how to use the GPU accelerator delegate
with these APIs with TensorFlow Lite with Google Play services.
[TensorFlow Lite with Google Play services](../play_services) is the recommended
path to use TensorFlow Lite on Android. If your application is targeting devices
not running Google Play, see the
[GPU with Interpreter API and standalone TensorFlow Lite](#standalone)
section.
### Add project dependencies
To enable access to the GPU delegate, add
`com.google.android.gms:play-services-tflite-gpu` to your app's `build.gradle`
file:
```
dependencies {
...
implementation 'com.google.android.gms:play-services-tflite-java:16.4.0'
implementation 'com.google.android.gms:play-services-tflite-gpu:16.4.0'
}
```
### Enable GPU acceleration
Then initialize TensorFlow Lite with Google Play services with the GPU support:
<div>
<devsite-selector>
<section>
<h3>Kotlin</h3>
<p><pre class="prettyprint lang-kotlin">
val useGpuTask = TfLiteGpu.isGpuDelegateAvailable(context)
val interpreterTask = useGpuTask.continueWith { useGpuTask ->
TfLite.initialize(context,
TfLiteInitializationOptions.builder()
.setEnableGpuDelegateSupport(useGpuTask.result)
.build())
}
</pre></p>
</section>
<section>
<h3>Java</h3>
<p><pre class="prettyprint lang-java">
Task<boolean> useGpuTask = TfLiteGpu.isGpuDelegateAvailable(context);
Task<Options> interpreterOptionsTask = useGpuTask.continueWith({ task ->
TfLite.initialize(context,
TfLiteInitializationOptions.builder()
.setEnableGpuDelegateSupport(true)
.build());
});
</pre></p>
</section>
</devsite-selector>
</div>
You can finally initialize the interpreter passing a `GpuDelegateFactory`
through `InterpreterApi.Options`:
<div>
<devsite-selector>
<section>
<h3>Kotlin</h3>
<p><pre class="prettyprint lang-kotlin">
val options = InterpreterApi.Options()
.setRuntime(TfLiteRuntime.FROM_SYSTEM_ONLY)
.addDelegateFactory(GpuDelegateFactory())
val interpreter = InterpreterApi(model, options)
// Run inference
writeToInput(input)
interpreter.run(input, output)
readFromOutput(output)
</pre></p>
</section>
<section>
<h3>Java</h3>
<p><pre class="prettyprint lang-java">
Options options = InterpreterApi.Options()
.setRuntime(TfLiteRuntime.FROM_SYSTEM_ONLY)
.addDelegateFactory(new GpuDelegateFactory());
Interpreter interpreter = new InterpreterApi(model, options);
// Run inference
writeToInput(input);
interpreter.run(input, output);
readFromOutput(output);
</pre></p>
</section>
</devsite-selector>
</div>
Note: The GPU delegate must be created on the same thread that runs it.
Otherwise, you may see the following error, `TfLiteGpuDelegate Invoke:
GpuDelegate must run on the same thread where it was initialized.`
The GPU delegate can also be used with ML model binding in Android Studio.
For more information, see
[Generate model interfaces using metadata](../../inference_with_metadata/codegen#acceleration).
## Use GPU with standalone TensorFlow Lite {:#standalone}
If your application is targets devices which are not running Google Play,
it is possible to bundle the GPU delegate to your application and use it
with the standalone version of TensorFlow Lite.
### Add project dependencies
To enable access to the GPU delegate, add
`org.tensorflow:tensorflow-lite-gpu-delegate-plugin` to your app's `build.gradle`
file:
```
dependencies {
...
implementation 'org.tensorflow:tensorflow-lite'
implementation 'org.tensorflow:tensorflow-lite-gpu-delegate-plugin'
}
```
### Enable GPU acceleration
Then run TensorFlow Lite on GPU with `TfLiteDelegate`. In Java, you can specify
the `GpuDelegate` through `Interpreter.Options`.
<div>
<devsite-selector>
<section>
<h3>Kotlin</h3>
<p><pre class="prettyprint lang-kotlin">
import org.tensorflow.lite.Interpreter
import org.tensorflow.lite.gpu.CompatibilityList
import org.tensorflow.lite.gpu.GpuDelegate
val compatList = CompatibilityList()
val options = Interpreter.Options().apply{
if(compatList.isDelegateSupportedOnThisDevice){
// if the device has a supported GPU, add the GPU delegate
val delegateOptions = compatList.bestOptionsForThisDevice
this.addDelegate(GpuDelegate(delegateOptions))
} else {
// if the GPU is not supported, run on 4 threads
this.setNumThreads(4)
}
}
val interpreter = Interpreter(model, options)
// Run inference
writeToInput(input)
interpreter.run(input, output)
readFromOutput(output)
</pre></p>
</section>
<section>
<h3>Java</h3>
<p><pre class="prettyprint lang-java">
import org.tensorflow.lite.Interpreter;
import org.tensorflow.lite.gpu.CompatibilityList;
import org.tensorflow.lite.gpu.GpuDelegate;
// Initialize interpreter with GPU delegate
Interpreter.Options options = new Interpreter.Options();
CompatibilityList compatList = CompatibilityList();
if(compatList.isDelegateSupportedOnThisDevice()){
// if the device has a supported GPU, add the GPU delegate
GpuDelegate.Options delegateOptions = compatList.getBestOptionsForThisDevice();
GpuDelegate gpuDelegate = new GpuDelegate(delegateOptions);
options.addDelegate(gpuDelegate);
} else {
// if the GPU is not supported, run on 4 threads
options.setNumThreads(4);
}
Interpreter interpreter = new Interpreter(model, options);
// Run inference
writeToInput(input);
interpreter.run(input, output);
readFromOutput(output);
</pre></p>
</section>
</devsite-selector>
</div>
### Quantized models {:#quantized-models}
Android GPU delegate libraries support quantized models by default. You do not
have to make any code changes to use quantized models with the GPU delegate. The
following section explains how to disable quantized support for testing or
experimental purposes.
#### Disable quantized model support
The following code shows how to ***disable*** support for quantized models.
<div>
<devsite-selector>
<section>
<h3>Java</h3>
<p><pre class="prettyprint lang-java">
GpuDelegate delegate = new GpuDelegate(new GpuDelegate.Options().setQuantizedModelsAllowed(false));
Interpreter.Options options = (new Interpreter.Options()).addDelegate(delegate);
</pre></p>
</section>
</devsite-selector>
</div>
For more information about running quantized models with GPU acceleration,
see [GPU delegate](../../performance/gpu#quantized-models) overview.
tensorflow/lite/g3doc/android/delegates/gpu_native.md
+180
View File
@@ -0,0 +1,180 @@
# GPU acceleration delegate with C/C++ API
Using graphics processing units (GPUs) to run your machine learning (ML) models
can dramatically improve the performance and the user experience of your
ML-enabled applications. On Android devices, you can enable GPU-accelerated
execution of your models using a
[*delegate*](https://ai.google.dev/edge/litert/performance/delegates) and one of
the following APIs:
- Interpreter API - [guide](./gpu)
- Task library API - [guide](./gpu_task.md)
- Native (C/C++) API - this guide
This guide covers advanced uses of the GPU delegate for the C API, C++ API, and
use of quantized models. For more information about using the GPU delegate for
TensorFlow Lite, including best practices and advanced techniques, see the
[GPU delegates](https://ai.google.dev/edge/litert/performance/gpu) page.
## Enable GPU acceleration
Use the TensorFlow Lite GPU delegate for Android in C or C++ by creating the
delegate with `TfLiteGpuDelegateV2Create()` and destroying it with
`TfLiteGpuDelegateV2Delete()`, as shown in the following example code:
```c++
// Set up interpreter.
auto model = FlatBufferModel::BuildFromFile(model_path);
if (!model) return false;
ops::builtin::BuiltinOpResolver op_resolver;
InterpreterBuilder builder(*model, op_resolver);
// NEW: Prepare GPU delegate.
auto* delegate = TfLiteGpuDelegateV2Create(/*default options=*/nullptr);
builder.AddDelegate(delegate);
// Build interpreter.
std::unique_ptr<Interpreter> interpreter;
if (builder(&interpreter) != kTfLiteOk) return false;
// Run inference.
WriteToInputTensor(interpreter->typed_input_tensor<float>(0));
if (interpreter->Invoke() != kTfLiteOk) return false;
ReadFromOutputTensor(interpreter->typed_output_tensor<float>(0));
// NEW: Clean up.
TfLiteGpuDelegateV2Delete(delegate);
```
Review the `TfLiteGpuDelegateOptionsV2` object code to build a delegate instance
with custom options. You can initialize the default options with
`TfLiteGpuDelegateOptionsV2Default()` and then modify them as necessary.
The TensorFlow Lite GPU delegate for Android in C or C++ uses the
[Bazel](https://bazel.io) build system. You can build the delegate using the
following command:
```sh
bazel build -c opt --config android_arm64 tensorflow/lite/delegates/gpu:delegate # for static library
bazel build -c opt --config android_arm64 tensorflow/lite/delegates/gpu:libtensorflowlite_gpu_delegate.so # for dynamic library
```
When calling `InterpreterBuilder::operator()` (e.g. `builder(&interpreter)`),
`Interpreter::ModifyGraphWithDelegate()`, or
`Interpreter::Invoke()`, the caller must have an `EGLContext` in the current
thread and `Interpreter::Invoke()` must be called from the same `EGLContext`. If
an `EGLContext` does not exist, the delegate creates one internally, but then
you must ensure that `Interpreter::Invoke()` is always called from the same
thread in which `InterpreterBuilder::operator()` or
`Interpreter::ModifyGraphWithDelegate` was called.
#### With TensorFlow Lite in Google Play Services:
If you are using TensorFlow Lite in Google Play Services
[C API](https://ai.google.dev/edge/litert/android/native), you’ll need to use
the Java/Kotlin API to check if a GPU delegate is available for your device
before initializing the TensorFlow Lite runtime.
Add the GPU delegate gradle dependencies to your application:
```
implementation 'com.google.android.gms:play-services-tflite-gpu:16.4.0'
```
Then, check the GPU availability and initialize TfLiteNative if the check is
successful:
<div>
<devsite-selector>
<section>
<h3>Java</h3>
<pre class="prettyprint">
Task<Void> tfLiteHandleTask =
TfLiteGpu.isGpuDelegateAvailable(this)
.onSuccessTask(gpuAvailable -> {
TfLiteInitializationOptions options =
TfLiteInitializationOptions.builder()
.setEnableGpuDelegateSupport(gpuAvailable).build();
return TfLiteNative.initialize(this, options);
}
);
</pre>
</section>
<section>
<h3>Kotlin</h3>
<pre class="prettyprint">
val tfLiteHandleTask = TfLiteGpu.isGpuDelegateAvailable(this)
.onSuccessTask { gpuAvailable ->
val options = TfLiteInitializationOptions.Builder()
.setEnableGpuDelegateSupport(gpuAvailable)
.build()
TfLiteNative.initialize(this, options)
}
</pre>
</section>
</devsite-selector>
</div>
You also need to update your CMake configuration to include the
`TFLITE_USE_OPAQUE_DELEGATE` compiler flag:
```
add_compile_definitions(TFLITE_USE_OPAQUE_DELEGATE)
```
The [FlatBuffers](https://flatbuffers.dev/) library is used to configure
delegate plugins, so you need to add it to the dependencies of your native code.
You can use the official `CMake` project configuration as follow:
```
target_include_directories(tflite-jni PUBLIC
third_party/headers # flatbuffers
...)
```
You can also just bundle the headers to your app.
Finally to use GPU inference in your C code, create the GPU delegate using
`TFLiteSettings`:
```
#include "flatbuffers/flatbuffers.h"
#include "tensorflow/lite/acceleration/configuration/configuration_generated.h"
flatbuffers::FlatBufferBuilder fbb;
tflite::TFLiteSettingsBuilder builder(fbb);
const tflite::TFLiteSettings* tflite_settings =
flatbuffers::GetTemporaryPointer(fbb, builder.Finish());
const TfLiteOpaqueDelegatePlugin* pluginCApi = TfLiteGpuDelegatePluginCApi();
TfLiteOpaqueDelegate* gpu_delegate = pluginCApi->create(tflite_settings);
```
## Quantized models {:#quantized-models}
Android GPU delegate libraries support quantized models by default. You do not
have to make any code changes to use quantized models with the GPU delegate. The
following section explains how to disable quantized support for testing or
experimental purposes.
#### Disable quantized model support
The following code shows how to ***disable*** support for quantized models.
<div>
<devsite-selector>
<section>
<h3>C++</h3>
<p><pre class="prettyprint lang-c++">
TfLiteGpuDelegateOptionsV2 options = TfLiteGpuDelegateOptionsV2Default();
options.experimental_flags = TFLITE_GPU_EXPERIMENTAL_FLAGS_NONE;
auto* delegate = TfLiteGpuDelegateV2Create(&options);
builder.AddDelegate(delegate);
</pre></p>
</section>
</devsite-selector>
</div>
For more information about running quantized models with GPU acceleration, see
[GPU delegate](https://ai.google.dev/edge/litert/performance/gpu#quantized_models)
overview.
tensorflow/lite/g3doc/android/delegates/gpu_task.md
+164
View File
@@ -0,0 +1,164 @@
# GPU acceleration delegate with Task library
Using graphics processing units (GPUs) to run your machine learning (ML) models
can dramatically improve the performance and the user experience
of your ML-enabled applications. On Android devices, you can enable
GPU-accelerated execution of your models using a
[*delegate*](../../performance/delegates) and one of the following APIs:
- Interpreter API - [guide](./gpu)
- Task library API - this guide
- Native (C/C++) API - this [guide](./gpu_native)
This page describes how to enable GPU acceleration for TensorFlow Lite models in
Android apps using the Task library.
For more information about the GPU delegate for TensorFlow Lite,
including best practices and advanced techniques, see the
[GPU delegates](../../performance/gpu) page.
## Use GPU with TensorFlow Lite with Google Play services
The TensorFlow Lite
[Task Libraries](../../inference_with_metadata/task_library/overview) provide a
set of task-specific APIs for building machine learning applications. This
section describes how to use the GPU accelerator delegate with these APIs using
TensorFlow Lite with Google Play services.
[TensorFlow Lite with Google Play services](../play_services) is the recommended
path to use TensorFlow Lite on Android. If your application is targeting devices
not running Google Play, see the
[GPU with Task Library and standalone TensorFlow Lite](#standalone)
section.
### Add project dependencies
To enable access to the GPU delegate with the TensorFlow Lite Task
Libraries using Google Play services, add
`com.google.android.gms:play-services-tflite-gpu` to the
dependencies of your app's `build.gradle` file:
```
dependencies {
...
implementation 'com.google.android.gms:play-services-tflite-gpu:16.4.0'
}
```
### Enable GPU acceleration
Then, verify asynchronously that GPU delegate is available for the device using
the
[`TfLiteGpu`](https://developers.google.com/android/reference/com/google/android/gms/tflite/gpu/support/TfLiteGpu)
class and enable the GPU delegate option for your Task API model class with the
[`BaseOptions`](https://www.tensorflow.org/lite/api_docs/java/org/tensorflow/lite/task/core/BaseOptions.Builder)
class. For example, you can set up GPU in `ObjectDetector` as shown in the
following code examples:
<div>
<devsite-selector>
<section>
<h3>Kotlin</h3>
<p><pre class="prettyprint lang-kotlin">
val useGpuTask = TfLiteGpu.isGpuDelegateAvailable(context)
lateinit val optionsTask = useGpuTask.continueWith { task ->
val baseOptionsBuilder = BaseOptions.builder()
if (task.result) {
baseOptionsBuilder.useGpu()
}
ObjectDetectorOptions.builder()
.setBaseOptions(baseOptionsBuilder.build())
.setMaxResults(1)
.build()
}
</pre></p>
</section>
<section>
<h3>Java</h3>
<p><pre class="prettyprint lang-java">
Task<Boolean> useGpuTask = TfLiteGpu.isGpuDelegateAvailable(context);
Task<ObjectDetectorOptions> optionsTask = useGpuTask.continueWith({ task ->
BaseOptions baseOptionsBuilder = BaseOptions.builder();
if (task.getResult()) {
baseOptionsBuilder.useGpu();
}
return ObjectDetectorOptions.builder()
.setBaseOptions(baseOptionsBuilder.build())
.setMaxResults(1)
.build()
});
</pre></p>
</section>
</devsite-selector>
</div>
## Use GPU with standalone TensorFlow Lite {:#standalone}
If your application is targets devices which are not running Google Play,
it is possible to bundle the GPU delegate to your application and use it
with the standalone version of TensorFlow Lite.
### Add project dependencies
To enable access to the GPU delegate with the TensorFlow Lite Task
Libraries using the standalone version of TensorFlow Lite, add
`org.tensorflow:tensorflow-lite-gpu-delegate-plugin` to the
dependencies of your app's `build.gradle` file:
```
dependencies {
...
implementation 'org.tensorflow:tensorflow-lite'
implementation 'org.tensorflow:tensorflow-lite-gpu-delegate-plugin'
}
```
### Enable GPU acceleration
Then enable the GPU delegate option for your Task API model class with the
[`BaseOptions`](https://www.tensorflow.org/lite/api_docs/java/org/tensorflow/lite/task/core/BaseOptions.Builder)
class. For example, you can set up GPU in `ObjectDetector` as shown in the
following code examples:
<div>
<devsite-selector>
<section>
<h3>Kotlin</h3>
<p><pre class="prettyprint lang-kotlin">
import org.tensorflow.lite.task.core.BaseOptions
import org.tensorflow.lite.task.gms.vision.detector.ObjectDetector
val baseOptions = BaseOptions.builder().useGpu().build()
val options =
ObjectDetector.ObjectDetectorOptions.builder()
.setBaseOptions(baseOptions)
.setMaxResults(1)
.build()
val objectDetector = ObjectDetector.createFromFileAndOptions(
context, model, options)
</pre></p>
</section>
<section>
<h3>Java</h3>
<p><pre class="prettyprint lang-java">
import org.tensorflow.lite.task.core.BaseOptions
import org.tensorflow.lite.task.gms.vision.detector.ObjectDetector
BaseOptions baseOptions = BaseOptions.builder().useGpu().build();
ObjectDetectorOptions options =
ObjectDetectorOptions.builder()
.setBaseOptions(baseOptions)
.setMaxResults(1)
.build();
val objectDetector = ObjectDetector.createFromFileAndOptions(
context, model, options);
</pre></p>
</section>
</devsite-selector>
</div>
tensorflow/lite/g3doc/android/delegates/hexagon.md
+312
View File
@@ -0,0 +1,312 @@
# TensorFlow Lite Hexagon delegate
<aside class="warning">
<p><b>Warning:</b> The
<a href="https://www.tensorflow.org/lite/android/delegates/nnapi">
NNAPI</a> and <a href="https://www.tensorflow.org/lite/android/delegates/hexagon">
Hexagon</a> delegates are deprecated and no longer supported by TensorFlow
Lite. For more information, see the
<a href="https://developer.android.com/ndk/guides/neuralnetworks/migration-guide">
NNAPI Migration Guide</a> and
<a href="https://www.tensorflow.org/lite/performance/delegates">TF Lite
delegates documentation</a>.</p>
</aside>
This document explains how to use the TensorFlow Lite Hexagon Delegate in your
application using the Java and/or C API. The delegate leverages the Qualcomm
Hexagon library to execute quantized kernels on the DSP. Note that the delegate
is intended to *complement* NNAPI functionality, particularly for devices where
NNAPI DSP acceleration is unavailable (e.g., on older devices, or devices that
don’t yet have a DSP NNAPI driver).
**Supported devices:**
Currently the following Hexagon architecture are supported, including but not
limited to:
* Hexagon 680
* SoC examples: Snapdragon 821, 820, 660
* Hexagon 682
* SoC examples: Snapdragon 835
* Hexagon 685
* SoC examples: Snapdragon 845, Snapdragon 710, QCS410, QCS610, QCS605,
QCS603
* Hexagon 690
* SoC examples: Snapdragon 855, RB5
**Supported models:**
The Hexagon delegate supports all models that conform to our
[8-bit symmetric quantization spec](https://www.tensorflow.org/lite/performance/quantization_spec),
including those generated using
[post-training integer quantization](https://www.tensorflow.org/lite/performance/post_training_integer_quant).
UInt8 models trained with the legacy
[quantization-aware training](https://github.com/tensorflow/tensorflow/tree/r1.13/tensorflow/contrib/quantize)
path are also supported, for example,
[these quantized versions](https://www.tensorflow.org/lite/guide/hosted_models#quantized_models)
on our Hosted Models page.
## Hexagon delegate Java API
```java
public class HexagonDelegate implements Delegate, Closeable {
/*
* Creates a new HexagonDelegate object given the current 'context'.
* Throws UnsupportedOperationException if Hexagon DSP delegation is not
* available on this device.
*/
public HexagonDelegate(Context context) throws UnsupportedOperationException
/**
* Frees TFLite resources in C runtime.
*
* User is expected to call this method explicitly.
*/
@Override
public void close();
}
```
### Example usage
#### Step 1. Edit app/build.gradle to use the nightly Hexagon delegate AAR
```
dependencies {
...
implementation 'org.tensorflow:tensorflow-lite:0.0.0-nightly-SNAPSHOT'
implementation 'org.tensorflow:tensorflow-lite-hexagon:0.0.0-nightly-SNAPSHOT'
}
```
#### Step 2. Add Hexagon libraries to your Android app {:#hexagon_versions}
* Download and run hexagon_nn_skel.run. It should provide 3 different shared
libraries “libhexagon_nn_skel.so”, “libhexagon_nn_skel_v65.so”,
“libhexagon_nn_skel_v66.so”
* [v1.10.3](https://storage.cloud.google.com/download.tensorflow.org/tflite/hexagon_nn_skel_1_10_3_1.run)
* [v1.14](https://storage.cloud.google.com/download.tensorflow.org/tflite/hexagon_nn_skel_v1.14.run)
* [v1.17](https://storage.cloud.google.com/download.tensorflow.org/tflite/hexagon_nn_skel_v1.17.0.0.run)
* [v1.20](https://storage.cloud.google.com/download.tensorflow.org/tflite/hexagon_nn_skel_v1.20.0.0.run)
* [v1.20.0.1](https://storage.cloud.google.com/download.tensorflow.org/tflite/hexagon_nn_skel_v1.20.0.1.run)
Caution: The currently released versions of the Hexagon delegate, up to version
1.20.0.1, are no longer supported. An updated version of this delegate is
expected soon.
Note: You must accept the license agreement before using the delegate.
Note: You must use the hexagon_nn libraries with the compatible version of
interface library. Interface library is part of the AAR and fetched by bazel
through the
[config](https://github.com/tensorflow/tensorflow/blob/master/third_party/hexagon/workspace.bzl)
The version in the bazel config is the version you should use.
* Include all 3 in your app with other shared libraries. See
[How to add shared library to your app](#how-to-add-shared-library-to-your-app).
The delegate will automatically pick the one with best performance depending
on the device.
Note: If your app will be built for both 32 and 64-bit ARM devices, then
add the Hexagon shared libs to both 32 and 64-bit lib folders.
#### Step 3. Create a delegate and initialize a TensorFlow Lite Interpreter
```java
import org.tensorflow.lite.HexagonDelegate;
// Create the Delegate instance.
try {
hexagonDelegate = new HexagonDelegate(activity);
tfliteOptions.addDelegate(hexagonDelegate);
} catch (UnsupportedOperationException e) {
// Hexagon delegate is not supported on this device.
}
tfliteInterpreter = new Interpreter(tfliteModel, tfliteOptions);
// Dispose after finished with inference.
tfliteInterpreter.close();
if (hexagonDelegate != null) {
hexagonDelegate.close();
}
```
## Hexagon delegate C API
```c
struct TfLiteHexagonDelegateOptions {
// This corresponds to the debug level in the Hexagon SDK. 0 (default)
// means no debug.
int debug_level;
// This corresponds to powersave_level in the Hexagon SDK.
// where 0 (default) means high performance which means more power
// consumption.
int powersave_level;
// If set to true, performance information about the graph will be dumped
// to Standard output, this includes cpu cycles.
// WARNING: Experimental and subject to change anytime.
bool print_graph_profile;
// If set to true, graph structure will be dumped to Standard output.
// This is usually beneficial to see what actual nodes executed on
// the DSP. Combining with 'debug_level' more information will be printed.
// WARNING: Experimental and subject to change anytime.
bool print_graph_debug;
};
// Return a delegate that uses Hexagon SDK for ops execution.
// Must outlive the interpreter.
TfLiteDelegate*
TfLiteHexagonDelegateCreate(const TfLiteHexagonDelegateOptions* options);
// Do any needed cleanup and delete 'delegate'.
void TfLiteHexagonDelegateDelete(TfLiteDelegate* delegate);
// Initializes the DSP connection.
// This should be called before doing any usage of the delegate.
// "lib_directory_path": Path to the directory which holds the
// shared libraries for the Hexagon NN libraries on the device.
void TfLiteHexagonInitWithPath(const char* lib_directory_path);
// Same as above method but doesn't accept the path params.
// Assumes the environment setup is already done. Only initialize Hexagon.
Void TfLiteHexagonInit();
// Clean up and switch off the DSP connection.
// This should be called after all processing is done and delegate is deleted.
Void TfLiteHexagonTearDown();
```
### Example usage
#### Step 1. Edit app/build.gradle to use the nightly Hexagon delegate AAR
```
dependencies {
...
implementation 'org.tensorflow:tensorflow-lite:0.0.0-nightly-SNAPSHOT'
implementation 'org.tensorflow:tensorflow-lite-hexagon:0.0.0-nightly-SNAPSHOT'
}
```
#### Step 2. Add Hexagon libraries to your Android app
* Download and run hexagon_nn_skel.run. It should provide 3 different shared
libraries “libhexagon_nn_skel.so”, “libhexagon_nn_skel_v65.so”,
“libhexagon_nn_skel_v66.so”
* [v1.10.3](https://storage.cloud.google.com/download.tensorflow.org/tflite/hexagon_nn_skel_1_10_3_1.run)
* [v1.14](https://storage.cloud.google.com/download.tensorflow.org/tflite/hexagon_nn_skel_v1.14.run)
* [v1.17](https://storage.cloud.google.com/download.tensorflow.org/tflite/hexagon_nn_skel_v1.17.0.0.run)
* [v1.20](https://storage.cloud.google.com/download.tensorflow.org/tflite/hexagon_nn_skel_v1.20.0.0.run)
* [v1.20.0.1](https://storage.cloud.google.com/download.tensorflow.org/tflite/hexagon_nn_skel_v1.20.0.1.run)
Caution: The currently released versions of the Hexagon delegate, up to version
1.20.0.1, are no longer supported. An updated version of this delegate is
expected soon.
Note: You must accept the license agreement before using the delegate.
Note: You must use the hexagon_nn libraries with the compatible version of
interface library. Interface library is part of the AAR and fetched by bazel
through the
[config](https://github.com/tensorflow/tensorflow/blob/master/third_party/hexagon/workspace.bzl).
The version in the bazel config is the version you should use.
* Include all 3 in your app with other shared libraries. See
[How to add shared library to your app](#how-to-add-shared-library-to-your-app).
The delegate will automatically pick the one with best performance depending
on the device.
Note: If your app will be built for both 32 and 64-bit ARM devices, then
add the Hexagon shared libs to both 32 and 64-bit lib folders.
#### Step 3. Include the C header
* The header file "hexagon_delegate.h" can be downloaded from
[GitHub](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/delegates/hexagon/hexagon_delegate.h)
or extracted from the Hexagon delegate AAR.
#### Step 4. Create a delegate and initialize a TensorFlow Lite Interpreter
* In your code, ensure the native Hexagon library is loaded. This can be done
by calling `System.loadLibrary("tensorflowlite_hexagon_jni");` \
in your Activity or Java entry-point.
* Create a delegate, example:
```c
#include "tensorflow/lite/delegates/hexagon/hexagon_delegate.h"
// Assuming shared libraries are under "/data/local/tmp/"
// If files are packaged with native lib in android App then it
// will typically be equivalent to the path provided by
// "getContext().getApplicationInfo().nativeLibraryDir"
const char[] library_directory_path = "/data/local/tmp/";
TfLiteHexagonInitWithPath(library_directory_path); // Needed once at startup.
::tflite::TfLiteHexagonDelegateOptions params = {0};
// 'delegate_ptr' Need to outlive the interpreter. For example,
// If your use case requires resizing the input or anything that can trigger
// re-applying delegates then 'delegate_ptr' must outlive the interpreter.
auto* delegate_ptr = ::tflite::TfLiteHexagonDelegateCreate(&params);
Interpreter::TfLiteDelegatePtr delegate(delegate_ptr,
[](TfLiteDelegate* delegate) {
::tflite::TfLiteHexagonDelegateDelete(delegate);
});
interpreter->ModifyGraphWithDelegate(delegate.get());
// After usage of delegate.
TfLiteHexagonTearDown(); // Needed once at end of app/DSP usage.
```
## Add the shared library to your app
* Create folder “app/src/main/jniLibs”, and create a directory for each target
architecture. For example,
* ARM 64-bit: `app/src/main/jniLibs/arm64-v8a`
* ARM 32-bit: `app/src/main/jniLibs/armeabi-v7a`
* Put your .so in the directory that match the architecture.
Note: If you're using App Bundle for publishing your Application, you might want
to set android.bundle.enableUncompressedNativeLibs=false in the
gradle.properties file.
## Feedback
For issues, please create a
[GitHub](https://github.com/tensorflow/tensorflow/issues/new?template=50-other-issues.md)
issue with all the necessary repro details, including the phone model and board
used (`adb shell getprop ro.product.device` and `adb shell getprop
ro.board.platform`).
## FAQ
* Which ops are supported by the delegate?
* See the current list of
[supported ops and constraints](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/delegates/hexagon/README.md)
* How can I tell that the model is using the DSP when I enable the delegate?
* Two log messages will be printed when you enable the delegate - one to
indicate if the delegate was created and another to indicate how many
nodes are running using the delegate. \
`Created TensorFlow Lite delegate for Hexagon.` \
`Hexagon delegate: X nodes delegated out of Y nodes.`
* Do I need all Ops in the model to be supported to run the delegate?
* No, the Model will be partitioned into subgraphs based on the supported
ops. Any unsupported ops will run on the CPU.
* How can I build the Hexagon delegate AAR from source?
* Use `bazel build -c opt --config=android_arm64
tensorflow/lite/delegates/hexagon/java:tensorflow-lite-hexagon`.
* Why does Hexagon delegate fail to initialize although my Android device has
a supported SoC?
* Verify if your device indeed has a supported SoC. Run `adb shell cat
/proc/cpuinfo | grep Hardware` and see if it returns something like
"Hardware : Qualcomm Technologies, Inc MSMXXXX".
* Some phone manufacturers use different SoCs for the same phone model.
Therefore, Hexagon delegate may only work on some but not all devices of
the same phone model.
* Some phone manufactures intentionally restrict the use of Hexagon DSP
from non-system Android apps, making the Hexagon delegate unable to
work.
* My phone has locked DSP access. I rooted the phone and still can't run the
delegate, what to do ?
* Make sure to disable SELinux enforce by running `adb shell setenforce 0`
tensorflow/lite/g3doc/android/delegates/nnapi.md
+264
View File
@@ -0,0 +1,264 @@
# TensorFlow Lite NNAPI delegate
<aside class="warning">
<p><b>Warning:</b> The
<a href="https://www.tensorflow.org/lite/android/delegates/nnapi">
NNAPI</a> and <a href="https://www.tensorflow.org/lite/android/delegates/hexagon">
Hexagon</a> delegates are deprecated and no longer supported by TensorFlow
Lite. For more information, see the
<a href="https://developer.android.com/ndk/guides/neuralnetworks/migration-guide">
NNAPI Migration Guide</a> and
<a href="https://www.tensorflow.org/lite/performance/delegates">TF Lite
delegates documentation</a>.</p>
</aside>
The
[Android Neural Networks API (NNAPI)](https://developer.android.com/ndk/guides/neuralnetworks)
is available on all Android devices running Android 8.1 (API level 27) or
higher. It provides acceleration for TensorFlow Lite models on Android devices
with supported hardware accelerators including:
* Graphics Processing Unit (GPU)
* Digital Signal Processor (DSP)
* Neural Processing Unit (NPU)
Performance will vary depending on the specific hardware available on device.
This page describes how to use the NNAPI delegate with the TensorFlow Lite
Interpreter in Java and Kotlin. For Android C APIs, please refer to
[Android Native Developer Kit documentation](https://developer.android.com/ndk/guides/neuralnetworks).
## Trying the NNAPI delegate on your own model
### Gradle import
The NNAPI delegate is part of the TensorFlow Lite Android interpreter, release
1.14.0 or higher. You can import it to your project by adding the following to
your module gradle file:
```groovy
dependencies {
implementation 'org.tensorflow:tensorflow-lite:+'
}
```
### Initializing the NNAPI delegate
Add the code to initialize the NNAPI delegate before you initialize the
TensorFlow Lite interpreter.
Note: Although NNAPI is supported from API Level 27 (Android Oreo MR1), the
support for operations improved significantly for API Level 28 (Android Pie)
onwards. As a result, we recommend developers use the NNAPI delegate for Android
Pie or above for most scenarios.
#### kotlin
```kotlin
import android.content.res.AssetManager
import org.tensorflow.lite.Interpreter
import org.tensorflow.lite.nnapi.NnApiDelegate
import java.io.FileInputStream
import java.io.IOException
import java.nio.MappedByteBuffer
import java.nio.channels.FileChannel
...
val options = Interpreter.Options()
var nnApiDelegate: NnApiDelegate? = null
// Initialize interpreter with NNAPI delegate for Android Pie or above
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.P) {
nnApiDelegate = NnApiDelegate()
options.addDelegate(nnApiDelegate)
}
val assetManager = assets
// Initialize TFLite interpreter
val tfLite: Interpreter
try {
tfLite = Interpreter(loadModelFile(assetManager, "model.tflite"), options)
} catch (e: Exception) {
throw RuntimeException(e)
}
// Run inference
// ...
// Unload delegate
tfLite.close()
nnApiDelegate?.close()
...
@Throws(IOException::class)
private fun loadModelFile(assetManager: AssetManager, modelFilename: String): MappedByteBuffer {
val fileDescriptor = assetManager.openFd(modelFilename)
val inputStream = FileInputStream(fileDescriptor.fileDescriptor)
val fileChannel = inputStream.channel
val startOffset = fileDescriptor.startOffset
val declaredLength = fileDescriptor.declaredLength
return fileChannel.map(FileChannel.MapMode.READ_ONLY, startOffset, declaredLength)
}
...
```
#### java
```java
import android.content.res.AssetManager;
import org.tensorflow.lite.Interpreter;
import org.tensorflow.lite.nnapi.NnApiDelegate;
import java.io.FileInputStream;
import java.io.IOException;
import java.nio.MappedByteBuffer;
import java.nio.channels.FileChannel;
...
Interpreter.Options options = (new Interpreter.Options());
NnApiDelegate nnApiDelegate = null;
// Initialize interpreter with NNAPI delegate for Android Pie or above
if(Build.VERSION.SDK_INT >= Build.VERSION_CODES.P) {
nnApiDelegate = new NnApiDelegate();
options.addDelegate(nnApiDelegate);
}
AssetManager assetManager = getAssets();
// Initialize TFLite interpreter
try {
tfLite = new Interpreter(loadModelFile(assetManager, "model.tflite"), options);
} catch (Exception e) {
throw new RuntimeException(e);
}
// Run inference
// ...
// Unload delegate
tfLite.close();
if(null != nnApiDelegate) {
nnApiDelegate.close();
}
...
private MappedByteBuffer loadModelFile(AssetManager assetManager, String modelFilename) throws IOException {
AssetFileDescriptor fileDescriptor = assetManager.openFd(modelFilename);
FileInputStream inputStream = new FileInputStream(fileDescriptor.getFileDescriptor());
FileChannel fileChannel = inputStream.getChannel();
long startOffset = fileDescriptor.getStartOffset();
long declaredLength = fileDescriptor.getDeclaredLength();
return fileChannel.map(FileChannel.MapMode.READ_ONLY, startOffset, declaredLength);
}
...
```
## Best practices
### Test performance before deploying
Runtime performance can vary significantly due to model architecture, size,
operations, hardware availability, and runtime hardware utilization. For
example, if an app heavily utilizes the GPU for rendering, NNAPI acceleration
may not improve performance due to resource contention. We recommend running a
simple performance test using the debug logger to measure inference time. Run
the test on several phones with different chipsets (manufacturer or models from
the same manufacturer) that are representative of your user base before enabling
NNAPI in production.
For advanced developers, TensorFlow Lite also offers
[a model benchmark tool for Android](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/tools/benchmark).
### Create a device exclusion list
In production, there may be cases where NNAPI does not perform as expected. We
recommend developers maintain a list of devices that should not use NNAPI
acceleration in combination with particular models. You can create this list
based on the value of `"ro.board.platform"`, which you can retrieve using the
following code snippet:
```java
String boardPlatform = "";
try {
Process sysProcess =
new ProcessBuilder("/system/bin/getprop", "ro.board.platform").
redirectErrorStream(true).start();
BufferedReader reader = new BufferedReader
(new InputStreamReader(sysProcess.getInputStream()));
String currentLine = null;
while ((currentLine=reader.readLine()) != null){
boardPlatform = line;
}
sysProcess.destroy();
} catch (IOException e) {}
Log.d("Board Platform", boardPlatform);
```
For advanced developers, consider maintaining this list via a remote
configuration system. The TensorFlow team is actively working on ways to
simplify and automate discovering and applying the optimal NNAPI configuration.
### Quantization
Quantization reduces model size by using 8-bit integers or 16-bit floats instead
of 32-bit floats for computation. 8-bit integer model sizes are a quarter of the
32-bit float versions; 16-bit floats are half of the size. Quantization can
improve performance significantly though the process could trade off some model
accuracy.
There are multiple types of post-training quantization techniques available,
but, for maximum support and acceleration on current hardware, we recommend
[full integer quantization](post_training_quantization#full_integer_quantization_of_weights_and_activations).
This approach converts both the weight and the operations into integers. This
quantization process requires a representative dataset to work.
### Use supported models and ops
If the NNAPI delegate does not support some of the ops or parameter combinations
in a model, the framework only runs the supported parts of the graph on the
accelerator. The remainder runs on the CPU, which results in split execution.
Due to the high cost of CPU/accelerator synchronization, this may result in
slower performance than executing the whole network on the CPU alone.
NNAPI performs best when models only use
[supported ops](https://developer.android.com/ndk/guides/neuralnetworks#model).
The following models are known to be compatible with NNAPI:
* [MobileNet v1 (224x224) image classification (float model download)](https://ai.googleblog.com/2017/06/mobilenets-open-source-models-for.html)
[(quantized model download)](http://download.tensorflow.org/models/mobilenet_v1_2018_08_02/mobilenet_v1_1.0_224_quant.tgz)
\
_(image classification model designed for mobile and embedded based vision
applications)_
* [MobileNet v2 SSD object detection](https://ai.googleblog.com/2018/07/accelerated-training-and-inference-with.html)
[(download)](https://storage.googleapis.com/download.tensorflow.org/models/tflite/gpu/mobile_ssd_v2_float_coco.tflite)
\
_(image classification model that detects multiple objects with bounding
boxes)_
* [MobileNet v1(300x300) Single Shot Detector (SSD) object detection](https://ai.googleblog.com/2018/07/accelerated-training-and-inference-with.html)
[(download)] (https://storage.googleapis.com/download.tensorflow.org/models/tflite/coco_ssd_mobilenet_v1_1.0_quant_2018_06_29.zip)
* [PoseNet for pose estimation](https://github.com/tensorflow/tfjs-models/tree/master/posenet)
[(download)](https://storage.googleapis.com/download.tensorflow.org/models/tflite/gpu/multi_person_mobilenet_v1_075_float.tflite)
\
_(vision model that estimates the poses of a person(s) in image or video)_
NNAPI acceleration is also not supported when the model contains
dynamically-sized outputs. In this case, you will get a warning like:
```none
ERROR: Attempting to use a delegate that only supports static-sized tensors \
with a graph that has dynamic-sized tensors.
```
### Enable NNAPI CPU implementation
A graph that can't be processed completely by an accelerator can fall back to
the NNAPI CPU implementation. However, since this is typically less performant
than the TensorFlow interpreter, this option is disabled by default in the NNAPI
delegate for Android 10 (API Level 29) or above. To override this behavior, set
`setUseNnapiCpu` to `true` in the `NnApiDelegate.Options` object.
tensorflow/lite/g3doc/android/development.md
+227
View File
@@ -0,0 +1,227 @@
# Development tools for Android
TensorFlow Lite provides a number of tools for integrating models into Android
apps. This page describes development tools for use in building apps with
Kotlin, Java, and C++, as well as support for TensorFlow Lite development in
Android Studio.
Key Point: In general, you should use the
[TensorFlow Lite Task Library](#tensorflow-lite-task-library-task_library) for
integrating TensorFlow Lite into your Android app, unless your use case is not
supported by that library. If it's not supported by the Task Library, use the
[TensorFlow Lite library](#tensorflow-lite-library-lite_lib) and
[Support library](#tensorflow-lite-support-library-support_lib).
To get started quickly writing Android code, see the
[Quickstart for Android](../android/quickstart.md)
## Tools for building with Kotlin and Java
The following sections describe development tools for TensorFlow Lite that use
the Kotlin and Java languages.
### TensorFlow Lite Task Library {:#task_library}
TensorFlow Lite Task Library contains a set of powerful and easy-to-use
task-specific libraries for app developers to build with TensorFlow Lite.
It provides optimized out-of-box model interfaces for popular machine learning
tasks, such as image classification, question and answer, etc. The model
interfaces are specifically designed for each task to achieve the best
performance and usability. Task Library works cross-platform and is supported on
Java and C++.
To use the Task Library in your Android app, use the AAR from MavenCentral for
[Task Vision library](https://search.maven.org/artifact/org.tensorflow/tensorflow-lite-task-vision)
,
[Task Text library](https://search.maven.org/artifact/org.tensorflow/tensorflow-lite-task-text)
and
[Task Audio Library](https://search.maven.org/artifact/org.tensorflow/tensorflow-lite-task-audio)
, respectively.
You can specify this in your `build.gradle` dependencies as follows:
```build
dependencies {
implementation 'org.tensorflow:tensorflow-lite-task-vision:+'
implementation 'org.tensorflow:tensorflow-lite-task-text:+'
implementation 'org.tensorflow:tensorflow-lite-task-audio:+'
}
```
If you use nightly snapshots, make sure you add the
[Sonatype snapshot repository](./lite_build.md#use_nightly_snapshots) to your
project.
See the introduction in the
[TensorFlow Lite Task Library overview](../inference_with_metadata/task_library/overview.md)
for more details.
### TensorFlow Lite library {:#lite_lib}
Use the TensorFlow Lite library in your Android app by adding the
[AAR hosted at MavenCentral](https://search.maven.org/artifact/org.tensorflow/tensorflow-lite)
to your development project.
You can specify this in your `build.gradle` dependencies as follows:
```build
dependencies {
implementation 'org.tensorflow:tensorflow-lite:+'
}
```
If you use nightly snapshots, make sure you add the
[Sonatype snapshot repository](./lite_build.md#use_nightly_snapshots) to your
project.
This AAR includes binaries for all of the
[Android ABIs](https://developer.android.com/ndk/guides/abis). You can reduce
the size of your application's binary by only including the ABIs you need to
support.
Unless you are targeting specific hardware, you should omit the `x86`, `x86_64`,
and `arm32` ABIs in most cases. You can configure this with the following Gradle
configuration. It specifically includes only `armeabi-v7a` and `arm64-v8a`, and
should cover most modern Android devices.
```build
android {
defaultConfig {
ndk {
abiFilters 'armeabi-v7a', 'arm64-v8a'
}
}
}
```
To learn more about `abiFilters`, see
[Android ABIs](https://developer.android.com/ndk/guides/abis)
in the Android NDK documentation.
### TensorFlow Lite Support Library {:#support_lib}
The TensorFlow Lite Android Support Library makes it easier to integrate models
into your application. It provides high-level APIs that help transform raw input
data into the form required by the model, and interpret the model's output,
reducing the amount of boilerplate code required.
It supports common data formats for inputs and outputs, including images and
arrays. It also provides pre- and post-processing units that perform tasks such
as image resizing and cropping.
Use the Support Library in your Android app by including the TensorFlow Lite
[Support Library AAR hosted at MavenCentral](https://search.maven.org/artifact/org.tensorflow/tensorflow-lite-support).
You can specify this in your `build.gradle` dependencies as follows:
```build
dependencies {
implementation 'org.tensorflow:tensorflow-lite-support:+'
}
```
If you use nightly snapshots, make sure you add the
[Sonatype snapshot repository](./lite_build.md#use_nightly_snapshots) to your
project.
For instructions on how to get started, see the
[TensorFlow Lite Android Support Library](../inference_with_metadata/lite_support.md).
### Minimum Android SDK versions for libraries
| Library | `minSdkVersion` | Device Requirements |
| --------------------------- | --------------- | ---------------------- |
| tensorflow-lite | 19 | NNAPI usage requires |
: : : API 27+ :
| tensorflow-lite-gpu | 19 | GLES 3.1 or OpenCL |
: : : (typically only :
: : : available on API 21+ :
| tensorflow-lite-hexagon | 19 | - |
| tensorflow-lite-support | 19 | - |
| tensorflow-lite-task-vision | 21 | android.graphics.Color |
: : : related API requires :
: : : API 26+ :
| tensorflow-lite-task-text | 21 | - |
| tensorflow-lite-task-audio | 23 | - |
| tensorflow-lite-metadata | 19 | - |
### Using Android Studio
In addition to the development libraries described above, Android Studio
also provides support for integrating TensorFlow Lite models, as described
below.
#### Android Studio ML Model Binding
The ML Model Binding feature of Android Studio 4.1 and later allows you to
import `.tflite` model files into your existing Android app, and generate
interface classes to make it easier to integrate your code with a model.
To import a TensorFlow Lite (TFLite) model:
1. Right-click on the module you would like to use the TFLite model or click on
**File > New > Other > TensorFlow Lite Model**.
1. Select the location of your TensorFlow Lite file. Note that the tooling
configures the module's dependency with ML Model binding and automatically
adds all required dependencies to your Android module's `build.gradle` file.
Note: Select the second checkbox for importing TensorFlow GPU if you want to
use [GPU acceleration](../performance/gpu.md).
1. Click `Finish` to begin the import process. When the import is finished, the
tool displays a screen describing the model, including its input and output
tensors.
1. To start using the model, select Kotlin or Java, copy and paste the code in
the **Sample Code** section.
You can return to the model information screen by double clicking the TensorFlow
Lite model under the `ml` directory in Android Studio. For more information on
using the Modle Binding feature of Android Studio, see the Android Studio
[release notes](https://developer.android.com/studio/releases#4.1-tensor-flow-lite-models).
For an overview of using model binding in Android Studio, see the code example
[instructions](https://github.com/tensorflow/examples/blob/master/lite/examples/image_classification/android/README.md).
## Tools for building with C and C++
The C and C++ libraries for TensorFlow Lite are primarily intended for
developers using the Android Native Development Kit (NDK) to build their apps.
There are two ways to use TFLite through C++ if you build your app with the NDK:
### TFLite C API
Using this API is the *recommended* approach for developers using the NDK.
Download the
[TensorFlow Lite AAR hosted at MavenCentral](https://search.maven.org/artifact/org.tensorflow/tensorflow/tensorflow-lite)
file, rename to `tensorflow-lite-*.zip`, and unzip it. You must include the four
header files in the `headers/tensorflow/lite/` and `headers/tensorflow/lite/c/`
folders and the relevant `libtensorflowlite_jni.so` dynamic library in the `jni/`
folder in your NDK project.
The `c_api.h` header file contains basic documentation about using the TFLite C
API.
### TFLite C++ API
If you want to use TFLite through C++ API, you can build the C++ shared
libraries:
32bit armeabi-v7a:
```sh
bazel build -c opt --config=android_arm //tensorflow/lite:libtensorflowlite.so
```
64bit arm64-v8a:
```sh
bazel build -c opt --config=android_arm64 //tensorflow/lite:libtensorflowlite.so
```
Currently, there is no straightforward way to extract all header files needed,
so you must include all header files in `tensorflow/lite/` from the TensorFlow
repository. Additionally, you will need header files from
[FlatBuffers](https://github.com/google/flatbuffers) and
[Abseil](https://github.com/abseil/abseil-cpp).
tensorflow/lite/g3doc/android/index.md
+281
View File
@@ -0,0 +1,281 @@
# TensorFlow Lite for Android
TensorFlow Lite lets you run TensorFlow machine learning (ML) models in your
Android apps. The TensorFlow Lite system provides prebuilt and customizable
execution environments for running models on Android quickly and efficiently,
including options for hardware acceleration.
## Learning roadmap {:.hide-from-toc}
<section class="devsite-landing-row devsite-landing-row-3-up devsite-landing-row-100" header-position="top">
<div class="devsite-landing-row-inner">
<div class="devsite-landing-row-group">
<div class="devsite-landing-row-item devsite-landing-row-item-no-media tfo-landing-page-card" description-position="bottom">
<div class="devsite-landing-row-item-description">
<div class="devsite-landing-row-item-body">
<div class="devsite-landing-row-item-description-content">
<a href="#machine_learning_models">
<h3 class="no-link hide-from-toc" id="code-design" data-text="Code design">Code design</h3></a>
Learn concepts and code design for building Android apps with TensorFlow
Lite, just <a href="#machine_learning_models">keep reading</a>.
</div>
</div>
</div>
</div>
<div class="devsite-landing-row-item devsite-landing-row-item-no-media tfo-landing-page-card" description-position="bottom">
<div class="devsite-landing-row-item-description">
<div class="devsite-landing-row-item-body">
<div class="devsite-landing-row-item-description-content">
<a href="./quickstart">
<h3 class="no-link hide-from-toc" id="coding-quickstart" data-text="Coding Quickstart">Code Quickstart</h3></a>
Start coding an Android app with TensorFlow Lite right away with the
<a href="./quickstart">Quickstart</a>.
</div>
</div>
</div>
</div>
<div class="devsite-landing-row-item devsite-landing-row-item-no-media tfo-landing-page-card" description-position="bottom">
<div class="devsite-landing-row-item-description">
<div class="devsite-landing-row-item-body">
<div class="devsite-landing-row-item-description-content">
<a href="../models">
<h3 class="no-link hide-from-toc" id="ml-models" data-text="ML models">ML models</h3></a>
Learn about choosing and using ML models with TensorFlow Lite, see the
<a href="../models">Models</a> docs.
</div>
</div>
</div>
</div>
</div>
</div>
</section>
## Machine learning models
TensorFlow Lite uses TensorFlow models that are converted into a smaller,
portable, more efficient machine learning model format. You can use pre-built
models with TensorFlow Lite on Android, or build your own TensorFlow models and
convert them to TensorFlow Lite format.
**Key Point:** TensorFlow Lite models and TensorFlow models have a *different
format and are not interchangeable.* TensorFlow models can be converted into the
TensorFlow Lite models, but that process is not reversible.
This page discusses using already-built machine learning models and does not
cover building, training, testing, or converting models. Learn more about
picking, modifying, building, and converting machine learning models for
TensorFlow Lite in the [Models](../models) section.
## Run models on Android
A TensorFlow Lite model running inside an Android app takes in data, processes
the data, and generates a prediction based on the model's logic. A TensorFlow
Lite model requires a special runtime environment in order to execute, and the
data that is passed into the model must be in a specific data format, called a
[*tensor*](../../guide/tensor). When a model processes the data, known as running
an *inference*, it generates prediction results as new tensors, and passes them
to the Android app so it can take action, such as showing the result to a user
or executing additional business logic.
![Functional execution flow for TensorFlow Lite models in Android
apps](../../images/lite/android/tf_execution_flow_android.png)
**Figure 1.** Functional execution flow for TensorFlow Lite models in Android
apps.
At the functional design level, your Android app needs the following elements to
run a TensorFlow Lite model:
- TensorFlow Lite **runtime environment** for executing the model
- **Model input handler** to transform data into tensors
- **Model output handler** to receive output result tensors and interpret them
as prediction results
The following sections describe how the TensorFlow Lite libraries and tools
provide these functional elements.
## Build apps with TensorFlow Lite
This section describes the recommended, most common path for implementing
TensorFlow Lite in your Android App. You should pay most attention to the
[runtime environment](#runtime) and [development
libraries](#apis) sections. If you have developed a custom
model, make sure to review the [Advanced development
paths](#adv_development) section.
### Runtime environment options {:#runtime}
There are several ways you can enable a runtime environment for executing models
in your Android app. These are the preferred options:
- **TensorFlow Lite in
[Google Play services runtime environment](./play_services) (recommended)**
- Stand-alone TensorFlow Lite runtime environment
In general, you should use the runtime environment provided by Google Play
services because it is more space-efficient than the standard environment since
it loads dynamically, keeping your app size smaller. Google Play services also
automatically uses the most recent, stable release of the TensorFlow Lite
runtime, giving you additional features and improved performance over time. If
you offer your app on devices that do not include Google Play services or you
need to closely manage your ML runtime environment, then you should use the
standard TensorFlow Lite runtime. This option bundles additional code into your
app, allowing you to have more control over the ML runtime in your app at the
cost of increasing your app's download size.
You access these runtime environments in your Android app by adding TensorFlow
Lite development libraries to your app development environment. For information
about how to use the standard runtime environments in your app, see the next
section.
Note: Some advanced use cases may require customization of model runtime
environment, which are described in the
[Advanced runtime environments](#adv_runtime) section.
### Development APIs and libraries {:#apis}
There are two main APIs you can use to integrate TensorFlow Lite machine
learning models into your Android app:
* **[TensorFlow Lite Task API](../api_docs/java/org/tensorflow/lite/task/core/package-summary) (recommended)**
* [TensorFlow Lite Interpreter API](../api_docs/java/org/tensorflow/lite/InterpreterApi)
The
[Interpreter API](../api_docs/java/org/tensorflow/lite/InterpreterApi)
provides classes and methods for running inferences with existing TensorFlow
Lite models. The TensorFlow Lite
[Task API](../api_docs/java/org/tensorflow/lite/task/core/package-summary)
wraps the Interpreter API and provides a higher-level programming interface
for performing common machine learning tasks on handling visual, audio, and
text data. You should use the Task API unless you find it does not support
your specific use case.
#### Libraries
You can access the Task APIs or the Interpreter API using the
[Google Play services](./play_services#add_tensorflow_lite_to_your_app).
You can also use the stand-alone libraries for
[TensorFlow Lite Tasks](../inference_with_metadata/task_library/overview) or the
TensorFlow Lite [core](./development#lite_lib) and
[support](./development#support_lib) libraries in your Android app.
For programming details about using TensorFlow Lite libraries and runtime
environments, see [Development tools for Android](./development).
### Obtain models {:#models}
Running a model in an Android app requires a TensorFlow Lite-format model. You
can use prebuilt models or build one with TensorFlow and convert it to the Lite
format. For more information on obtaining models for your Android app, see the
TensorFlow Lite [Models](../models)
section.
### Handle input data {:#input_data}
Any data you pass into a ML model must be a tensor with a specific data
structure, often called the *shape* of the tensor. To process data with a model,
your app code must transform data from its native format, such as image, text,
or audio data, into a tensor in the required shape for your model.
**Note:** Many TensorFlow Lite models come with embedded
[metadata](../inference_with_metadata/overview)
that describes the required input data.
The
[TensorFlow Lite Task library](../inference_with_metadata/task_library/overview)
provides data handling logic for transforming visual, text, and audio data into
tensors with the correct shape to be processed by a TensorFlow Lite model.
### Run inferences {:#inferences}
Processing data through a model to generate a prediction result is known as
running an *inference*. Running an inference in an Android app requires a
TensorFlow Lite [runtime environment](#runtime), a
[model](#models) and [input data](#input_data).
The speed at which a model can generate an inference on a particular device
depends on the size of the data processed, the complexity of the model, and the
available computing resources such as memory and CPU, or specialized processors
called *accelerators*. Machine learning models can run faster on these
specialized processors such as graphics processing units (GPUs) and tensor
processing units (TPUs), using TensorFlow Lite hardware drivers called
*delegates*. For more information about delegates and hardware acceleration of
model processing, see the
[Hardware acceleration overview](../performance/delegates).
### Handle output results {:#output_results}
Models generate prediction results as tensors, which must be handled by your
Android app by taking action or displaying a result to the user. Model output
results can be as simple as a number corresponding to a single result (0 = dog,
1 = cat, 2 = bird) for an image classification, to much more complex results,
such as multiple bounding boxes for several classified objects in an image, with
prediction confidence ratings between 0 and 1.
**Note:** Many TensorFlow Lite models come with embedded
[metadata](../inference_with_metadata/overview)
that describes the output results of a model and how to interpret it.
## Advanced development paths {:#adv_development}
When using more sophisticated and customized TensorFlow Lite models, you may
need to use more advanced development approaches than what is described above.
The following sections describe advanced techniques for executing models and
developing them for TensorFlow Lite in Android apps.
### Advanced runtime environments {:#adv_runtime}
In addition to the standard runtime and Google Play services runtime
environments for TensorFlow Lite, there are additional runtime environments you
can use with your Android app. The most likely use for these environments is if
you have a machine learning model that uses ML operations that are not supported
by the standard runtime environment for TensorFlow Lite.
- [Flex runtime](../guide/ops_select) for TensorFlow Lite
- Custom-built TensorFlow Lite runtime
The TensorFlow Lite [Flex runtime](../guide/ops_select) allows you to include
specific operators required for your model. As an advanced option for running
your model, you can build TensorFlow Lite for Android to include operators and
other functionality required for running your TensorFlow machine learning model.
For more information, see [Build TensorFlow Lite for Android](./lite_build).
### C and C++ APIs
TensorFlow Lite also provides an API for running models using C and C++. If your
app uses the [Android NDK](https://developer.android.com/ndk), you should
consider using this API. You may also want to consider using this API if you
want to be able to share code between multiple platforms. For more information
about this development option, see the
[Development tools](./development#tools_for_building_with_c_and_c) page.
### Server-based model execution
In general, you should run models in your app on an Android device to take
advantage of lower latency and improved data privacy for your users. However,
there are cases where running a model on a cloud server, off device, is a better
solution. For example, if you have a large model which does not easily compress
down to a size that fits on your users' Android devices, or can be executed with
reasonable performance on those devices. This approach may also be your
preferred solution if consistent performance of the model across a wide range of
devices is top priority.
Google Cloud offers a full suite of services for running TensorFlow machine
learning models. For more information, see Google Cloud's [AI and machine
learning products](https://cloud.google.com/products/ai) page.
### Custom model development and optimization
More advanced development paths are likely to include developing custom machine
learning models and optimizing those models for use on Android devices. If you
plan to build custom models, make sure you consider applying
[quantization techniques](../performance/post_training_quantization)
to models to reduce memory and processing costs. For more information on how to
build high-performance models for use with TensorFlow Lite, see
[Performance best practices](../performance/best_practices)
in the Models section.
## Next Steps
- Try the Android [Quickstart](./quickstart) or tutorials
- Explore the TensorFlow Lite [examples](../examples)
- Learn how to find or build [TensorFlow Lite models](../models)
tensorflow/lite/g3doc/android/java.md
+556
View File
@@ -0,0 +1,556 @@
# TensorFlow Lite in Google Play services Java API
TensorFlow Lite in Google Play services can also be accessed using Java APIs, in
addition to the Native API. In particular, TensorFlow Lite in Google Play
services is available through the
[TensorFlow Lite Task API](https://www.tensorflow.org/lite/api_docs/java/org/tensorflow/lite/task/core/package-summary)
and the
[TensorFlow Lite Interpreter API](https://www.tensorflow.org/lite/api_docs/java/org/tensorflow/lite/InterpreterApi).
The Task Library provides optimized out-of-the-box model interfaces for common
machine learning tasks using visual, audio, and text data. The TensorFlow Lite
Interpreter API, provided by the TensorFlow runtime, provides a more
general-purpose interface for building and running ML models.
The following sections provide instructions on how to use the Interpreter and
Task Library APIs with TensorFlow Lite in Google Play services. While it is
possible for an app to use both the Interpreter APIs and Task Library APIs, most
apps should only use one set of APIs.
### Using the Task Library APIs
The TensorFlow Lite Task API wraps the Interpreter API and provides a high-level
programming interface for common machine learning tasks that use visual, audio,
and text data. You should use the Task API if your application requires one of
the
[supported tasks](../inference_with_metadata/task_library/overview.md#supported-tasks).
#### 1. Add project dependencies
Your project dependency depends on your machine learning use case. The Task APIs
contain the following libraries:
* Vision library: `org.tensorflow:tensorflow-lite-task-vision-play-services`
* Audio library: `org.tensorflow:tensorflow-lite-task-audio-play-services`
* Text library: `org.tensorflow:tensorflow-lite-task-text-play-services`
Add one of the dependencies to your app project code to access the Play services
API for TensorFlow Lite. For example, use the following to implement a vision
task:
```
dependencies {
...
implementation 'org.tensorflow:tensorflow-lite-task-vision-play-services:0.4.2'
...
}
```
Caution: The TensorFlow Lite Tasks Audio library version 0.4.2 maven repository
is incomplete. Use version 0.4.2.1 for this library instead:
`org.tensorflow:tensorflow-lite-task-audio-play-services:0.4.2.1`.
#### 2. Add initialization of TensorFlow Lite
Initialize the TensorFlow Lite component of the Google Play services API
*before* using the TensorFlow Lite APIs. The following example initializes the
vision library:
<div>
<devsite-selector>
<section>
<h3>Kotlin</h3>
<pre class="prettyprint">
init {
TfLiteVision.initialize(context)
}
</pre>
</section>
</devsite-selector>
</div>
Important: Make sure the `TfLite.initialize` task completes before executing
code that accesses TensorFlow Lite APIs.
Tip: The TensorFlow Lite modules are installed at the same time your application
is installed or updated from the Play Store. You can check the availability of
the modules by using `ModuleInstallClient` from the Google Play services APIs.
For more information on checking module availability, see
[Ensuring API availability with ModuleInstallClient](https://developers.google.com/android/guides/module-install-apis).
#### 3. Run inferences
After initializing the TensorFlow Lite component, call the `detect()` method to
generate inferences. The exact code within the `detect()` method varies
depending on the library and use case. The following is for a simple object
detection use case with the `TfLiteVision` library:
<div>
<devsite-selector>
<section>
<h3>Kotlin</h3>
<pre class="prettyprint">
fun detect(...) {
if (!TfLiteVision.isInitialized()) {
Log.e(TAG, "detect: TfLiteVision is not initialized yet")
return
}
if (objectDetector == null) {
setupObjectDetector()
}
...
}
</pre>
</section>
</devsite-selector>
</div>
Depending on the data format, you may also need to preprocess and convert your
data within the `detect()` method before generating inferences. For example,
image data for an object detector requires the following:
```kotlin
val imageProcessor = ImageProcessor.Builder().add(Rot90Op(-imageRotation / 90)).build()
val tensorImage = imageProcessor.process(TensorImage.fromBitmap(image))
val results = objectDetector?.detect(tensorImage)
```
### Using the Interpreter APIs
The Interpreter APIs offer more control and flexibility than the Task Library
APIs. You should use the Interpreter APIs if your machine learning task is not
supported by the Task library, or if you require a more general-purpose
interface for building and running ML models.
#### 1. Add project dependencies
Add the following dependencies to your app project code to access the Play
services API for TensorFlow Lite:
```
dependencies {
...
// Tensorflow Lite dependencies for Google Play services
implementation 'com.google.android.gms:play-services-tflite-java:16.4.0'
// Optional: include Tensorflow Lite Support Library
implementation 'com.google.android.gms:play-services-tflite-support:16.4.0'
...
}
```
#### 2. Add initialization of TensorFlow Lite
Initialize the TensorFlow Lite component of the Google Play services API
*before* using the TensorFlow Lite APIs:
<div>
<devsite-selector>
<section>
<h3>Kotlin</h3>
<pre class="prettyprint">
val initializeTask: Task&lt;Void> by lazy { TfLite.initialize(this) }
</pre>
</section>
<section>
<h3>Java</h3>
<pre class="prettyprint">
Task&lt;Void> initializeTask = TfLite.initialize(context);
</pre>
</section>
</devsite-selector>
</div>
Note: Make sure the `TfLite.initialize` task completes before executing code
that accesses TensorFlow Lite APIs. Use the `addOnSuccessListener()` method, as
shown in the next section.
#### 3. Create an Interpreter and set runtime option {:#step_3_interpreter}
Create an interpreter using `InterpreterApi.create()` and configure it to use
Google Play services runtime, by calling `InterpreterApi.Options.setRuntime()`,
as shown in the following example code:
<div>
<devsite-selector>
<section>
<h3>Kotlin</h3>
<pre class="prettyprint">
import org.tensorflow.lite.InterpreterApi
import org.tensorflow.lite.InterpreterApi.Options.TfLiteRuntime
...
private lateinit var interpreter: InterpreterApi
...
initializeTask.addOnSuccessListener {
val interpreterOption =
InterpreterApi.Options().setRuntime(TfLiteRuntime.FROM_SYSTEM_ONLY)
interpreter = InterpreterApi.create(
modelBuffer,
interpreterOption
)}
.addOnFailureListener { e ->
Log.e("Interpreter", "Cannot initialize interpreter", e)
}
</pre>
</section>
<section>
<h3>Java</h3>
<pre class="prettyprint">
import org.tensorflow.lite.InterpreterApi
import org.tensorflow.lite.InterpreterApi.Options.TfLiteRuntime
...
private InterpreterApi interpreter;
...
initializeTask.addOnSuccessListener(a -> {
interpreter = InterpreterApi.create(modelBuffer,
new InterpreterApi.Options().setRuntime(TfLiteRuntime.FROM_SYSTEM_ONLY));
})
.addOnFailureListener(e -> {
Log.e("Interpreter", String.format("Cannot initialize interpreter: %s",
e.getMessage()));
});
</pre>
</section>
</devsite-selector>
</div>
You should use the implementation above because it avoids blocking the Android
user interface thread. If you need to manage thread execution more closely, you
can add a `Tasks.await()` call to interpreter creation:
<div>
<devsite-selector>
<section>
<h3>Kotlin</h3>
<pre class="prettyprint">
import androidx.lifecycle.lifecycleScope
...
lifecycleScope.launchWhenStarted { // uses coroutine
initializeTask.await()
}
</pre>
</section>
<section>
<h3>Java</h3>
<pre class="prettyprint">
@BackgroundThread
InterpreterApi initializeInterpreter() {
Tasks.await(initializeTask);
return InterpreterApi.create(...);
}
</pre>
</section>
</devsite-selector>
</div>
Warning: Do not call `.await()` on the foreground user interface thread because
it interrupts display of user interface elements and creates a poor user
experience.
#### 4. Run inferences
Using the `interpreter` object you created, call the `run()` method to generate
an inference.
<div>
<devsite-selector>
<section>
<h3>Kotlin</h3>
<pre class="prettyprint">
interpreter.run(inputBuffer, outputBuffer)
</pre>
</section>
<section>
<h3>Java</h3>
<pre class="prettyprint">
interpreter.run(inputBuffer, outputBuffer);
</pre>
</section>
</devsite-selector>
</div>
## Hardware acceleration {:#hardware-acceleration}
TensorFlow Lite allows you to accelerate the performance of your model using
specialized hardware processors, such as graphics processing units (GPUs). You
can take advantage of these specialized processors using hardware drivers called
[*delegates*](https://www.tensorflow.org/lite/performance/delegates). You can
use the following hardware acceleration delegates with TensorFlow Lite in Google
Play services:
- *[GPU delegate](https://www.tensorflow.org/lite/performance/gpu)
(recommended)* - This delegate is provided through Google Play services and
is dynamically loaded, just like the Play services versions of the Task API
and Interpreter API.
- [*NNAPI delegate*](https://www.tensorflow.org/lite/android/delegates/nnapi) -
This delegate is available as an included library dependency in your Android
development project, and is bundled into your app.
For more information about hardware acceleration with TensorFlow Lite, see the
[TensorFlow Lite Delegates](https://www.tensorflow.org/lite/performance/delegates)
page.
### Checking device compatibility
Not all devices support GPU hardware acceleration with TFLite. In order to
mitigate errors and potential crashes, use the
`TfLiteGpu.isGpuDelegateAvailable` method to check whether a device is
compatible with the GPU delegate.
Use this method to confirm whether a device is compatible with GPU, and use CPU
or the NNAPI delegate as a fallback for when GPU is not supported.
```
useGpuTask = TfLiteGpu.isGpuDelegateAvailable(context)
```
Once you have a variable like `useGpuTask`, you can use it to determine whether
devices use the GPU delegate. The following examples show how this can be done
with both the Task Library and Interpreter APIs.
**With the Task Api**
<div>
<devsite-selector>
<section>
<h3>Kotlin</h3>
<pre class="prettyprint">
lateinit val optionsTask = useGpuTask.continueWith { task ->
val baseOptionsBuilder = BaseOptions.builder()
if (task.result) {
baseOptionsBuilder.useGpu()
}
ObjectDetectorOptions.builder()
.setBaseOptions(baseOptionsBuilder.build())
.setMaxResults(1)
.build()
}
</pre>
</section>
<section>
<h3>Java</h3>
<pre class="prettyprint">
Task&lt;ObjectDetectorOptions> optionsTask = useGpuTask.continueWith({ task ->
BaseOptions baseOptionsBuilder = BaseOptions.builder();
if (task.getResult()) {
baseOptionsBuilder.useGpu();
}
return ObjectDetectorOptions.builder()
.setBaseOptions(baseOptionsBuilder.build())
.setMaxResults(1)
.build()
});
</pre>
</section>
</devsite-selector>
</div>
**With the Interpreter Api**
<div>
<devsite-selector>
<section>
<h3>Kotlin</h3>
<pre class="prettyprint">
val interpreterTask = useGpuTask.continueWith { task ->
val interpreterOptions = InterpreterApi.Options()
.setRuntime(TfLiteRuntime.FROM_SYSTEM_ONLY)
if (task.result) {
interpreterOptions.addDelegateFactory(GpuDelegateFactory())
}
InterpreterApi.create(FileUtil.loadMappedFile(context, MODEL_PATH), interpreterOptions)
}
</pre>
</section>
<section>
<h3>Java</h3>
<pre class="prettyprint">
Task&lt;InterpreterApi.Options> interpreterOptionsTask = useGpuTask.continueWith({ task ->
InterpreterApi.Options options =
new InterpreterApi.Options().setRuntime(TfLiteRuntime.FROM_SYSTEM_ONLY);
if (task.getResult()) {
options.addDelegateFactory(new GpuDelegateFactory());
}
return options;
});
</pre>
</section>
</devsite-selector>
</div>
### GPU with Task Library APIs
To use the GPU delegate with the Task APIs:
1. Update the project dependencies to use the GPU delegate from Play services:
```
implementation 'com.google.android.gms:play-services-tflite-gpu:16.4.0'
```
1. Initialize the GPU delegate with `setEnableGpuDelegateSupport`. For example,
you can initialize the GPU delegate for `TfLiteVision` with the following:
<div>
<devsite-selector>
<section>
<h3>Kotlin</h3>
<pre class="prettyprint">
TfLiteVision.initialize(context, TfLiteInitializationOptions.builder().setEnableGpuDelegateSupport(true).build())
</pre>
</section>
<section>
<h3>Java</h3>
<pre class="prettyprint">
TfLiteVision.initialize(context, TfLiteInitializationOptions.builder().setEnableGpuDelegateSupport(true).build());
</pre>
</section>
</devsite-selector>
</div>
1. Enable the GPU delegate option with
[`BaseOptions`](https://www.tensorflow.org/lite/api_docs/java/org/tensorflow/lite/task/core/BaseOptions.Builder):
<div>
<devsite-selector>
<section>
<h3>Kotlin</h3>
<pre class="prettyprint">
val baseOptions = BaseOptions.builder().useGpu().build()
</pre>
</section>
<section>
<h3>Java</h3>
<pre class="prettyprint">
BaseOptions baseOptions = BaseOptions.builder().useGpu().build();
</pre>
</section>
</devsite-selector>
</div>
1. Configure the options using `.setBaseOptions`. For example, you can set up
GPU in `ObjectDetector` with the following:
<div>
<devsite-selector>
<section>
<h3>Kotlin</h3>
<pre class="prettyprint">
val options =
ObjectDetectorOptions.builder()
.setBaseOptions(baseOptions)
.setMaxResults(1)
.build()
</pre>
</section>
<section>
<h3>Java</h3>
<pre class="prettyprint">
ObjectDetectorOptions options =
ObjectDetectorOptions.builder()
.setBaseOptions(baseOptions)
.setMaxResults(1)
.build();
</pre>
</section>
</devsite-selector>
</div>
### GPU with Interpreter APIs
To use the GPU delegate with the Interpreter APIs:
1. Update the project dependencies to use the GPU delegate from Play services:
```
implementation 'com.google.android.gms:play-services-tflite-gpu:16.4.0'
```
1. Enable the GPU delegate option in the TFlite initialization:
<div>
<devsite-selector>
<section>
<h3>Kotlin</h3>
<pre class="prettyprint">
TfLite.initialize(context,
TfLiteInitializationOptions.builder()
.setEnableGpuDelegateSupport(true)
.build())
</pre>
</section>
<section>
<h3>Java</h3>
<pre class="prettyprint">
TfLite.initialize(context,
TfLiteInitializationOptions.builder()
.setEnableGpuDelegateSupport(true)
.build());
</pre>
</section>
</devsite-selector>
</div>
1. Enable GPU delegate in the interpreter options: set the delegate factory to
GpuDelegateFactory by calling `addDelegateFactory()
within`InterpreterApi.Options()`:
<div>
<devsite-selector>
<section>
<h3>Kotlin</h3>
<pre class="prettyprint">
val interpreterOption = InterpreterApi.Options()
.setRuntime(TfLiteRuntime.FROM_SYSTEM_ONLY)
.addDelegateFactory(GpuDelegateFactory())
</pre>
</section>
<section>
<h3>Java</h3>
<pre class="prettyprint">
Options interpreterOption = InterpreterApi.Options()
.setRuntime(TfLiteRuntime.FROM_SYSTEM_ONLY)
.addDelegateFactory(new GpuDelegateFactory());
</pre>
</section>
</devsite-selector>
</div>
## Migrating from stand-alone TensorFlow Lite {:#migrating}
If you are planning to migrate your app from stand-alone TensorFlow Lite to the
Play services API, review the following additional guidance for updating your
app project code:
1. Review the [Limitations](#limitations) section of this page to ensure your
use case is supported.
2. Prior to updating your code, do performance and accuracy checks for your
models, particularly if you are using versions of TensorFlow Lite earlier
than version 2.1, so you have a baseline to compare against the new
implementation.
3. If you have migrated all of your code to use the Play services API for
TensorFlow Lite, you should remove the existing TensorFlow Lite *runtime
library* dependencies (entries with
<code>org.tensorflow:**tensorflow-lite**:*</code>) from your build.gradle
file so that you can reduce your app size.
4. Identify all occurrences of `new Interpreter` object creation in your code,
and modify each one so that it uses the InterpreterApi.create() call. The
new TfLite.initialize is asynchronous, which means in most cases it's not a
drop-in replacement: you must register a listener for when the call
completes. Refer to the code snippet in [Step 3](#step_3_interpreter) code.
5. Add `import org.tensorflow.lite.InterpreterApi;` and `import
org.tensorflow.lite.InterpreterApi.Options.TfLiteRuntime;` to any source
files using the `org.tensorflow.lite.Interpreter` or
`org.tensorflow.lite.InterpreterApi` classes.
6. If any of the resulting calls to `InterpreterApi.create()` have only a
single argument, append `new InterpreterApi.Options()` to the argument list.
7. Append `.setRuntime(TfLiteRuntime.FROM_SYSTEM_ONLY)` to the last argument of
any calls to `InterpreterApi.create()`.
8. Replace all other occurrences of the `org.tensorflow.lite.Interpreter` class
with `org.tensorflow.lite.InterpreterApi`.
If you want to use stand-alone TensorFlow Lite and the Play services API
side-by-side, you must use TensorFlow Lite 2.9 (or later). TensorFlow Lite 2.8
and earlier versions are not compatible with the Play services API version.
tensorflow/lite/g3doc/android/lite_build.md
+238
View File
@@ -0,0 +1,238 @@
# Build TensorFlow Lite for Android
This document describes how to build TensorFlow Lite Android library on your
own. Normally, you do not need to locally build TensorFlow Lite Android library.
If you just want to use it, see the
[Android quickstart](../android/quickstart.md) for more details on how to use
them in your Android projects.
## Use Nightly Snapshots
Warning: Support for nightly snapshots is currently broken (b/446167415).
To use nightly snapshots, add the following repo to your root Gradle build
config.
```build
allprojects {
repositories { // should be already there
mavenCentral() // should be already there
maven { // add this repo to use snapshots
name 'ossrh-snapshot'
url 'https://oss.sonatype.org/content/repositories/snapshots'
}
}
}
```
add nightly snapshots to dependencies (or edit as needed) to your build.gradle
```groovy
...
dependencies {
...
implementation 'org.tensorflow:tensorflow-lite:0.0.0-nightly-SNAPSHOT'
implementation 'org.tensorflow:tensorflow-lite-gpu:0.0.0-nightly-SNAPSHOT'
implementation 'org.tensorflow:tensorflow-lite-support:0.0.0-nightly-SNAPSHOT'
...
}
...
```
## Build TensorFlow Lite locally
In some cases, you might wish to use a local build of TensorFlow Lite. For
example, you may be building a custom binary that includes
[operations selected from TensorFlow](https://www.tensorflow.org/lite/guide/ops_select),
or you may wish to make local changes to TensorFlow Lite.
### Set up build environment using Docker
* Download the Docker file. By downloading the Docker file, you agree that the
following terms of service govern your use thereof:
*By clicking to accept, you hereby agree that all use of the Android Studio and
Android Native Development Kit will be governed by the Android Software
Development Kit License Agreement available at
https://developer.android.com/studio/terms (such URL may be updated or changed
by Google from time to time).*
<!-- mdformat off(devsite fails if there are line-breaks in templates) -->
{% dynamic if 'tflite-android-tos' in user.acknowledged_walls and request.tld != 'cn' %}
You can download the Docker file
<a href="https://raw.githubusercontent.com/tensorflow/tensorflow/master/tensorflow/lite/tools/tflite-android.Dockerfile">here</a>
{% dynamic else %} You must acknowledge the terms of service to download the
file.
<button class="button-blue devsite-acknowledgement-link" data-globally-unique-wall-id="tflite-android-tos">Acknowledge</button>
{% dynamic endif %}
<!-- mdformat on -->
* You can optionally change the Android SDK or NDK version. Put the downloaded
Docker file in an empty folder and build your docker image by running:
```shell
docker build . -t tflite-builder -f tflite-android.Dockerfile
```
* Start the docker container interactively by mounting your current folder to
/host_dir inside the container (note that /tensorflow_src is the TensorFlow
repository inside the container):
```shell
docker run -it -v $PWD:/host_dir tflite-builder bash
```
If you use PowerShell on Windows, replace "$PWD" with "pwd".
If you would like to use a TensorFlow repository on the host, mount that host
directory instead (-v hostDir:/host_dir).
* Once you are inside the container, you can run the following to download
additional Android tools and libraries (note that you may need to accept the
license):
```shell
sdkmanager \
"build-tools;${ANDROID_BUILD_TOOLS_VERSION}" \
"platform-tools" \
"platforms;android-${ANDROID_API_LEVEL}"
```
Now you should proceed to the [Configure WORKSPACE and .bazelrc](#configure_workspace_and_bazelrc) section to configure the build settings.
After you finish building the libraries, you can copy them to /host_dir
inside the container so that you can access them on the host.
### Set up build environment without Docker
#### Install Bazel and Android Prerequisites
Bazel is the primary build system for TensorFlow. To build with it, you must
have it and the Android NDK and SDK installed on your system.
1. Install the latest version of the [Bazel build system](https://bazel.build/versions/master/docs/install.html).
2. The Android NDK is required to build the native (C/C++) TensorFlow Lite
code. The current recommended version is 25b, which may be found
[here](https://developer.android.com/ndk/downloads/older_releases.html#ndk-25b-downloads).
3. The Android SDK and build tools may be obtained
[here](https://developer.android.com/tools/revisions/build-tools.html), or
alternatively as part of
[Android Studio](https://developer.android.com/studio/index.html). Build
tools API >= 23 is the recommended version for building TensorFlow Lite.
### Configure WORKSPACE and .bazelrc
This is a one-time configuration step that is required to build the TF Lite
libraries. Run the `./configure` script in the root TensorFlow checkout
directory, and answer "Yes" when the script asks to interactively configure the `./WORKSPACE`
for Android builds. The script will attempt to configure settings using the
following environment variables:
* `ANDROID_SDK_HOME`
* `ANDROID_SDK_API_LEVEL`
* `ANDROID_NDK_HOME`
* `ANDROID_NDK_API_LEVEL`
If these variables aren't set, they must be provided interactively in the script
prompt. Successful configuration should yield entries similar to the following
in the `.tf_configure.bazelrc` file in the root folder:
```shell
build --action_env ANDROID_NDK_HOME="/usr/local/android/android-ndk-r25b"
build --action_env ANDROID_NDK_API_LEVEL="21"
build --action_env ANDROID_BUILD_TOOLS_VERSION="30.0.3"
build --action_env ANDROID_SDK_API_LEVEL="30"
build --action_env ANDROID_SDK_HOME="/usr/local/android/android-sdk-linux"
```
### Build and install
Once Bazel is properly configured, you can build the TensorFlow Lite AAR from
the root checkout directory as follows:
```sh
bazel build -c opt --cxxopt=--std=c++17 --config=android_arm64 \
--fat_apk_cpu=x86,x86_64,arm64-v8a,armeabi-v7a \
--define=android_dexmerger_tool=d8_dexmerger \
--define=android_incremental_dexing_tool=d8_dexbuilder \
//tensorflow/lite/java:tensorflow-lite
```
This will generate an AAR file in `bazel-bin/tensorflow/lite/java/`. Note
that this builds a "fat" AAR with several different architectures; if you don't
need all of them, use the subset appropriate for your deployment environment.
You can build smaller AAR files targeting only a set of models as follows:
```sh
bash tensorflow/lite/tools/build_aar.sh \
--input_models=model1,model2 \
--target_archs=x86,x86_64,arm64-v8a,armeabi-v7a
```
Above script will generate the `tensorflow-lite.aar` file and optionally the
`tensorflow-lite-select-tf-ops.aar` file if one of the models is using
Tensorflow ops. For more details, please see the
[Reduce TensorFlow Lite binary size](../guide/reduce_binary_size.md) section.
#### Add AAR directly to project
Move the `tensorflow-lite.aar` file into a directory called `libs` in your
project. Modify your app's `build.gradle` file to reference the new directory
and replace the existing TensorFlow Lite dependency with the new local library,
e.g.:
```
allprojects {
repositories {
mavenCentral()
maven { // Only for snapshot artifacts
name 'ossrh-snapshot'
url 'https://oss.sonatype.org/content/repositories/snapshots'
}
flatDir {
dirs 'libs'
}
}
}
dependencies {
compile(name:'tensorflow-lite', ext:'aar')
}
```
#### Install AAR to local Maven repository
Execute the following command from your root checkout directory:
```sh
mvn install:install-file \
-Dfile=bazel-bin/tensorflow/lite/java/tensorflow-lite.aar \
-DgroupId=org.tensorflow \
-DartifactId=tensorflow-lite -Dversion=0.1.100 -Dpackaging=aar
```
In your app's `build.gradle`, ensure you have the `mavenLocal()` dependency and
replace the standard TensorFlow Lite dependency with the one that has support
for select TensorFlow ops:
```
allprojects {
repositories {
mavenCentral()
maven { // Only for snapshot artifacts
name 'ossrh-snapshot'
url 'https://oss.sonatype.org/content/repositories/snapshots'
}
mavenLocal()
}
}
dependencies {
implementation 'org.tensorflow:tensorflow-lite:0.1.100'
}
```
Note that the `0.1.100` version here is purely for the sake of
testing/development. With the local AAR installed, you can use the standard
[TensorFlow Lite Java inference APIs](../guide/inference.md) in your app code.
tensorflow/lite/g3doc/android/native.md
+207
View File
@@ -0,0 +1,207 @@
# TensorFlow Lite in Google Play services C API (Beta)
Beta: TensorFlow Lite in Google Play services C API is currently in Beta.
TensorFlow Lite in Google Play services runtime allows you to run machine
learning (ML) models without statically bundling TensorFlow Lite libraries into
your app. This guide provide instructions on how to use the C APIs for Google
Play services.
Before working with the TensorFlow Lite in Google Play services C API, make sure
you have the [CMake](https://cmake.org/) build tool installed.
## Update your build configuration
Add the following dependencies to your app project code to access the Play
services API for TensorFlow Lite:
```
implementation "com.google.android.gms:play-services-tflite-java:16.2.0-beta02"
```
Then, enable the
[Prefab](https://developer.android.com/build/dependencies#build-system-configuration)
feature to access the C API from your CMake script by updating the android block
of your module's build.gradle file:
```
buildFeatures {
prefab = true
}
```
You finally need to add the package `tensorflowlite_jni_gms_client` imported
from the AAR as a dependency in your CMake script:
```
find_package(tensorflowlite_jni_gms_client REQUIRED CONFIG)
target_link_libraries(tflite-jni # your JNI lib target
tensorflowlite_jni_gms_client::tensorflowlite_jni_gms_client
android # other deps for your target
log)
# Also add -DTFLITE_IN_GMSCORE -DTFLITE_WITH_STABLE_ABI
# to the C/C++ compiler flags.
add_compile_definitions(TFLITE_IN_GMSCORE)
add_compile_definitions(TFLITE_WITH_STABLE_ABI)
```
## Initialize the TensorFlow Lite runtime
Before calling the TensorFlow Lite Native API you must initialize the
`TfLiteNative` runtime in your Java/Kotlin code.
<div>
<devsite-selector>
<section>
<h3>Java</h3>
<pre class="prettyprint">
Task tfLiteInitializeTask = TfLiteNative.initialize(context);
</pre>
</section>
<section>
<h3>Kotlin</h3>
<pre class="prettyprint">
val tfLiteInitializeTask: Task<Void> = TfLiteNative.initialize(context)
</pre>
</section>
</devsite-selector>
</div>
Using the Google Play services Task API, `TfLiteNative.initialize`
asynchronously loads the TFLite runtime from Google Play services into your
application's runtime process. Use `addOnSuccessListener()` to make sure the
`TfLite.initialize()` task completes before executing code that accesses
TensorFlow Lite APIs. Once the task has completed successfully, you can invoke
all the available TFLite Native APIs.
## Native code implementation
To use TensorFlow Lite in Google Play services with your native code, you can do
one of the following:
- declare new JNI functions to call native functions from your Java code
- Call the TensorFlow Lite Native API from your existing native C code.
JNI functions:
You can declare a new JNI function to make the TensorFlow Lite runtime declared
in Java/Kotlin accessible to your native code as follow:
<div>
<devsite-selector>
<section>
<h3>Java</h3>
<pre class="prettyprint">
package com.google.samples.gms.tflite.c;
public class TfLiteJni {
static {
System.loadLibrary("tflite-jni");
}
public TfLiteJni() { /**/ };
public native void loadModel(AssetManager assetManager, String assetName);
public native float[] runInference(float[] input);
}
</pre>
</section>
<section>
<h3>Kotlin</h3>
<pre class="prettyprint">
package com.google.samples.gms.tflite.c
class TfLiteJni() {
companion object {
init {
System.loadLibrary("tflite-jni")
}
}
external fun loadModel(assetManager: AssetManager, assetName: String)
external fun runInference(input: FloatArray): FloatArray
}
</pre>
</section>
</devsite-selector>
</div>
Matching the following `loadModel` and `runInference` native functions:
```
#ifdef __cplusplus
extern "C" {
#endif
void Java_com_google_samples_gms_tflite_c_loadModel(
JNIEnv *env, jobject tflite_jni, jobject asset_manager, jstring asset_name){}
//...
}
jfloatArray Java_com_google_samples_gms_tflite_c_TfLiteJni_runInference(
JNIEnv* env, jobject tfliteJni, jfloatArray input) {
//...
}
#ifdef __cplusplus
} // extern "C".
#endif
```
You can then call your C functions from your Java/Kotlin code:
<div>
<devsite-selector>
<section>
<h3>Java</h3>
<pre class="prettyprint">
tfLiteHandleTask.onSuccessTask(unused -> {
TfLiteJni jni = new TfLiteJni();
jni.loadModel(getAssets(), "add.bin");
//...
});
</pre>
</section>
<section>
<h3>Kotlin</h3>
<pre class="prettyprint">
tfLiteHandleTask.onSuccessTask {
val jni = TfLiteJni()
jni.loadModel(assets, "add.bin")
// ...
}
</pre>
</section>
</devsite-selector>
</div>
### TensorFlow Lite in C code
Include the appropriate API header file to include the TfLite with Google Play
services API:
```
#include "tensorflow/lite/c/c_api.h"
```
You can then use the regular TensorFlow Lite C API:
```
auto model = TfLiteModelCreate(model_asset, model_asset_length);
// ...
auto options = TfLiteInterpreterOptionsCreate();
// ...
auto interpreter = TfLiteInterpreterCreate(model, options);
```
The TensorFlow Lite with Google Play services Native API headers provide the
same API as the regular
[TensorFlow Lite C API](https://www.tensorflow.org/lite/api_docs/c), excluding
features that are deprecated or experimental. For now the functions and types
from the `c_api.h`, `c_api_types.h` and `common.h` headers are available. Please
note that functions from the `c_api_experimental.h` header are not supported.
The documentation can be found
[online](https://www.tensorflow.org/lite/api_docs/c).
You can use functions specific to TensorFlow Lite with Google Play Services by
including `tflite.h`.
tensorflow/lite/g3doc/android/play_services.md
+101
View File
@@ -0,0 +1,101 @@
# TensorFlow Lite in Google Play services
TensorFlow Lite is available in Google Play services runtime for all Android
devices running the current version of Play services. This runtime allows you to
run machine learning (ML) models without statically bundling TensorFlow Lite
libraries into your app.
With the Google Play services API, you can reduce the size of your apps and gain
improved performance from the latest stable version of the libraries. TensorFlow
Lite in Google Play services is the recommended way to use TensorFlow Lite on
Android.
You can get started with the Play services runtime with the
[Quickstart](../android/quickstart), which provides a step-by-step guide to
implement a sample application. If you are already using stand-alone TensorFlow
Lite in your app, refer to the
[Migrating from stand-alone TensorFlow Lite](#migrating) section to update an
existing app to use the Play services runtime. For more information about Google
Play services, see the
[Google Play services](https://developers.google.com/android/guides/overview)
website.
<aside class="note"> <b>Terms:</b> By accessing or using TensorFlow Lite in
Google Play services APIs, you agree to the <a href="#tos">Terms of Service</a>.
Please read and understand all applicable terms and policies before accessing
the APIs. </aside>
## Using the Play services runtime
The TensorFlow Lite in Google Play services is available through the following
programming language apis:
- Java API - [see guide](../android/java)
- C API - [see guide](../android/native)
## Limitations
TensorFlow Lite in Google Play services has the following limitations:
* Support for hardware acceleration delegates is limited to the delegates
listed in the [Hardware acceleration](#hardware-acceleration) section. No
other acceleration delegates are supported.
* Experimental or deprecated TensorFlow Lite APIs, including custom ops, are
not supported.
## Support and feedback {:#support}
You can provide feedback and get support through the TensorFlow Issue Tracker.
Please report issues and support requests using the
[Issue template](https://github.com/tensorflow/tensorflow/issues/new?title=TensorFlow+Lite+in+Play+Services+issue&template=tflite-in-play-services.md)
for TensorFlow Lite in Google Play services.
## Terms of service {:#tos}
Use of TensorFlow Lite in Google Play services APIs is subject to the
[Google APIs Terms of Service](https://developers.google.com/terms/).
### Privacy and data collection
When you use TensorFlow Lite in Google Play services APIs, processing of the
input data, such as images, video, text, fully happens on-device, and TensorFlow
Lite in Google Play services APIs does not send that data to Google servers. As
a result, you can use our APIs for processing data that should not leave the
device.
The TensorFlow Lite in Google Play services APIs may contact Google servers from
time to time in order to receive things like bug fixes, updated models and
hardware accelerator compatibility information. The TensorFlow Lite in Google
Play services APIs also sends metrics about the performance and utilization of
the APIs in your app to Google. Google uses this metrics data to measure
performance, debug, maintain and improve the APIs, and detect misuse or abuse,
as further described in our
[Privacy Policy](https://policies.google.com/privacy).
**You are responsible for informing users of your app about Google's processing
of TensorFlow Lite in Google Play services APIs metrics data as required by
applicable law.**
Data we collect includes the following:
+ Device information (such as manufacturer, model, OS version and build) and
available ML hardware accelerators (GPU and DSP). Used for diagnostics and
usage analytics.
+ Device identifier used for diagnostics and usage analytics.
+ App information (package name, app version). Used for diagnostics and usage
analytics.
+ API configuration (such as which delegates are being used). Used for
diagnostics and usage analytics.
+ Event type (such as interpreter creation, inference). Used for diagnostics
and usage analytics.
+ Error codes. Used for diagnostics.
+ Performance metrics. Used for diagnostics.
## Next steps
For more information about implementing machine learning in your mobile
application with TensorFlow Lite, see the
[TensorFlow Lite Developer Guide](https://www.tensorflow.org/lite/guide). You
can find additional TensorFlow Lite models for image classification, object
detection, and other applications on the
[TensorFlow Hub](https://tfhub.dev/s?deployment-format=lite).
tensorflow/lite/g3doc/android/quickstart.md
+360
View File
@@ -0,0 +1,360 @@
# Quickstart for Android
This page shows you how to build an Android app with TensorFlow Lite to analyze
a live camera feed and identify objects. This machine learning use case is
called *object detection*. The example app uses the TensorFlow Lite
[Task library for vision](../inference_with_metadata/task_library/overview#supported_tasks)
via [Google Play services](./play_services) to enable execution of the object
detection machine learning model, which is the recommended approach for building
an ML application with TensorFlow Lite.
<aside class="note"> <b>Terms:</b> By accessing or using TensorFlow Lite in
Google Play services APIs, you agree to the <a href="./play_services#tos">Terms
of Service</a>. Please read and understand all applicable terms and policies
before accessing the APIs. </aside>
![Object detection animated demo](https://storage.googleapis.com/download.tensorflow.org/tflite/examples/obj_detection_cat.gif){: .attempt-right width="250px"}
## Setup and run the example
For the first part of this exercise, download the
[example code](https://github.com/tensorflow/examples/tree/master/lite/examples/object_detection/android_play_services)
from GitHub and run it using [Android Studio](https://developer.android.com/studio/).
The following sections of this document explore the relevant sections of the
code example, so you can apply them to your own Android apps. You need the
following versions of these tools installed:
* Android Studio 4.2 or higher
* Android SDK version 21 or higher
Note: This example uses the camera, so you should run it on a physical Android
device.
### Get the example code
Create a local copy of the example code so you can build and run it.
To clone and setup the example code:
1. Clone the git repository
<pre class="devsite-click-to-copy">
git clone https://github.com/tensorflow/examples.git
</pre>
2. Configure your git instance to use sparse checkout, so you have only
the files for the object detection example app:
<pre class="devsite-click-to-copy">
cd examples
git sparse-checkout init --cone
git sparse-checkout set lite/examples/object_detection/android_play_services
</pre>
### Import and run the project
Use Android Studio to create a project from the downloaded example code, build
the project, and run it.
To import and build the example code project:
1. Start [Android Studio](https://developer.android.com/studio).
1. From the Android Studio **Welcome** page, choose **Import Project**, or
select **File > New > Import Project**.
1. Navigate to the example code directory containing the build.gradle file
(`...examples/lite/examples/object_detection/android_play_services/build.gradle`)
and select that directory.
After you select this directory, Android Studio creates a new project and builds
it. When the build completes, the Android Studio displays a `BUILD SUCCESSFUL`
message in the **Build Output** status panel.
To run the project:
1. From Android Studio, run the project by selecting **Run > Run…** and
**MainActivity**.
1. Select an attached Android device with a camera to test the app.
## How the example app works
The example app uses pre-trained object detection model, such as
[mobilenetv1.tflite](https://tfhub.dev/tensorflow/lite-model/ssd_mobilenet_v1/1/metadata/2?lite-format=tflite),
in TensorFlow Lite format look for objects in a live video stream from an
Android device's camera. The code for this feature is primarily in these files:
* [ObjectDetectorHelper.kt](https://github.com/tensorflow/examples/blob/master/lite/examples/object_detection/android_play_services/app/src/main/java/org/tensorflow/lite/examples/objectdetection/ObjectDetectorHelper.kt) -
Initializes the runtime environment, enables hardware acceleration, and
runs the object detection ML model.
* [CameraFragment.kt](https://github.com/tensorflow/examples/blob/master/lite/examples/object_detection/android_play_services/app/src/main/java/org/tensorflow/lite/examples/objectdetection/fragments/CameraFragment.kt) -
Builds the camera image data stream, prepares data for the model, and
displays the object detection results.
Note: This example app uses the TensorFlow Lite
[Task Library](../inference_with_metadata/task_library/overview#supported_tasks),
which provides easy-to-use, task-specific APIs for performing common machine
learning operations. For apps with more specific needs and customized ML
functions, consider using the
[Interpreter API](https://www.tensorflow.org/lite/api_docs/java/org/tensorflow/lite/InterpreterApi).
The next sections show you the key components of these code files, so you can
modify an Android app to add this functionality.
## Build the app {:#build_app}
The following sections explain the key steps to build your own Android app and
run the model shown in the example app. These instructions use the example app
shown earlier as a reference point.
Note: To follow along with these instructions and build your own app, create a
[basic Android project](https://developer.android.com/studio/projects/create-project)
using Android Studio.
### Add project dependencies {:#add_dependencies}
In your basic Android app, add the project dependencies for running TensorFlow
Lite machine learning models and accessing ML data utility functions. These
utility functions convert data such as images into a tensor data format that can
be processed by a model.
The example app uses the TensorFlow Lite
[Task library for vision](../inference_with_metadata/task_library/overview#supported_tasks)
from [Google Play services](./play_services) to enable execution of the object
detection machine learning model. The following instructions explain how to add
the required library dependencies to your own Android app project.
To add module dependencies:
1. In the Android app module that uses TensorFlow Lite, update the module's
`build.gradle` file to include the following dependencies. In the example
code, this file is located here:
`...examples/lite/examples/object_detection/android_play_services/app/build.gradle`
```
...
dependencies {
...
// Tensorflow Lite dependencies
implementation 'org.tensorflow:tensorflow-lite-task-vision-play-services:0.4.2'
implementation 'com.google.android.gms:play-services-tflite-gpu:16.1.0'
...
}
```
1. In Android Studio, sync the project dependencies by selecting: **File >
Sync Project with Gradle Files**.
### Initialize Google Play services
When you use [Google Play services](./play_services) to run TensorFlow Lite
models, you must initialize the service before you can use it. If you want to
use hardware acceleration support with the service, such as GPU acceleration,
you also enable that support as part of this initialization.
To initialize TensorFlow Lite with Google Play services:
1. Create a `TfLiteInitializationOptions` object and modify it to enable GPU
support:
```
val options = TfLiteInitializationOptions.builder()
.setEnableGpuDelegateSupport(true)
.build()
```
1. Use the `TfLiteVision.initialize()` method to enable use of the Play
services runtime, and set a listener to verify that it loaded successfully:
```
TfLiteVision.initialize(context, options).addOnSuccessListener {
objectDetectorListener.onInitialized()
}.addOnFailureListener {
// Called if the GPU Delegate is not supported on the device
TfLiteVision.initialize(context).addOnSuccessListener {
objectDetectorListener.onInitialized()
}.addOnFailureListener{
objectDetectorListener.onError("TfLiteVision failed to initialize: "
+ it.message)
}
}
```
### Initialize the ML model interpreter
Initialize the TensorFlow Lite machine learning model interpreter by loading the
model file and setting model parameters. A TensorFlow Lite model includes a
`.tflite` file containing the model code. You should store your models in the
`src/main/assets` directory of your development project, for example:
```
.../src/main/assets/mobilenetv1.tflite`
```
Tip: Task library interpreter code automatically looks for models in the
`src/main/assets` directory if you do not specify a file path.
To initialize the model:
1. Add a `.tflite` model file to the `src/main/assets` directory of your
development project, such as [ssd_mobilenet_v1](https://tfhub.dev/tensorflow/lite-model/ssd_mobilenet_v1/1/metadata/2).
1. Set the `modelName` variable to specify your ML model's file name:
```
val modelName = "mobilenetv1.tflite"
```
1. Set the options for model, such as the prediction threshold and results set
size:
```
val optionsBuilder =
ObjectDetector.ObjectDetectorOptions.builder()
.setScoreThreshold(threshold)
.setMaxResults(maxResults)
```
1. Enable GPU acceleration with the options and allow the code to fail
gracefully if acceleration is not supported on the device:
```
try {
optionsBuilder.useGpu()
} catch(e: Exception) {
objectDetectorListener.onError("GPU is not supported on this device")
}
```
1. Use the settings from this object to construct a TensorFlow Lite
[`ObjectDetector`](https://www.tensorflow.org/lite/api_docs/java/org/tensorflow/lite/task/vision/detector/ObjectDetector#createFromFile(Context,%20java.lang.String))
object that contains the model:
```
objectDetector =
ObjectDetector.createFromFileAndOptions(
context, modelName, optionsBuilder.build())
```
For more information about using hardware acceleration delegates with TensorFlow
Lite, see [TensorFlow Lite Delegates](../performance/delegates).
### Prepare data for the model
You prepare data for interpretation by the model by transforming existing data
such as images into the [Tensor](../api_docs/java/org/tensorflow/lite/Tensor)
data format, so it can be processed by your model. The data in a Tensor must
have specific dimensions, or shape, that matches the format of data used to
train the model. Depending on the model you use, you may need to transform the
data to fit what the model expects. The example app uses an
[`ImageAnalysis`](https://developer.android.com/reference/androidx/camera/core/ImageAnalysis)
object to extract image frames from the camera subsystem.
To prepare data for processing by the model:
1. Build an `ImageAnalysis` object to extract images in the required format:
```
imageAnalyzer =
ImageAnalysis.Builder()
.setTargetAspectRatio(AspectRatio.RATIO_4_3)
.setTargetRotation(fragmentCameraBinding.viewFinder.display.rotation)
.setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
.setOutputImageFormat(OUTPUT_IMAGE_FORMAT_RGBA_8888)
.build()
...
```
1. Connect the analyzer to the camera subsystem and create a bitmap buffer
to contain the data received from the camera:
```
.also {
it.setAnalyzer(cameraExecutor) { image ->
if (!::bitmapBuffer.isInitialized) {
bitmapBuffer = Bitmap.createBitmap(
image.width,
image.height,
Bitmap.Config.ARGB_8888
)
}
detectObjects(image)
}
}
```
1. Extract the specific image data needed by the model, and pass
the image rotation information:
```
private fun detectObjects(image: ImageProxy) {
// Copy out RGB bits to the shared bitmap buffer
image.use { bitmapBuffer.copyPixelsFromBuffer(image.planes[0].buffer) }
val imageRotation = image.imageInfo.rotationDegrees
objectDetectorHelper.detect(bitmapBuffer, imageRotation)
}
```
1. Complete any final data transformations and add the image data to a
`TensorImage` object, as shown in the `ObjectDetectorHelper.detect()`
method of the example app:
```
val imageProcessor = ImageProcessor.Builder().add(Rot90Op(-imageRotation / 90)).build()
// Preprocess the image and convert it into a TensorImage for detection.
val tensorImage = imageProcessor.process(TensorImage.fromBitmap(image))
```
### Run predictions
Once you create a
[TensorImage](../api_docs/java/org/tensorflow/lite/support/image/TensorImage)
object with image data in the correct format, you can run the model against that
data to produce a prediction, or *inference*. In the example app, this code
is contained in the `ObjectDetectorHelper.detect()` method.
To run a the model and generate predictions from image data:
- Run the prediction by passing the image data to your predict function:
```
val results = objectDetector?.detect(tensorImage)
```
### Handle model output
After you run image data against the object detection model, it produces a list
of prediction results which your app code must handle by executing additional
business logic, displaying results to the user, or taking other actions. The
object detection model in the example app produces a list of predictions and
bounding boxes for the detected objects. In the example app, the prediction
results are passed to a listener object for further processing and display to
the user.
To handle model prediction results:
1. Use a listener pattern to pass results to your app code or user interface
objects. The example app uses this pattern to pass detection results from
the `ObjectDetectorHelper` object to the `CameraFragment` object:
```
objectDetectorListener.onResults( // instance of CameraFragment
results,
inferenceTime,
tensorImage.height,
tensorImage.width)
```
1. Act on the results, such as displaying the prediction to the user. The
example app draws an overlay on the `CameraPreview` object to show the result:
```
override fun onResults(
results: MutableList<Detection>?,
inferenceTime: Long,
imageHeight: Int,
imageWidth: Int
) {
activity?.runOnUiThread {
fragmentCameraBinding.bottomSheetLayout.inferenceTimeVal.text =
String.format("%d ms", inferenceTime)
// Pass necessary information to OverlayView for drawing on the canvas
fragmentCameraBinding.overlay.setResults(
results ?: LinkedList<Detection>(),
imageHeight,
imageWidth
)
// Force a redraw
fragmentCameraBinding.overlay.invalidate()
}
}
```
## Next steps
* Learn more about the
[Task Library APIs](../inference_with_metadata/task_library/overview#supported_tasks)
* Learn more about the
[Interpreter APIs](https://www.tensorflow.org/lite/api_docs/java/org/tensorflow/lite/InterpreterApi).
* Explore the uses of TensorFlow Lite in the [examples](../examples).
* Learn more about using and building machine learning models with TensorFlow
Lite in the [Models](../models) section.
* Learn more about implementing machine learning in your mobile application in
the [TensorFlow Lite Developer Guide](../guide).
tensorflow/lite/g3doc/android/tutorials/audio_classification.md
+402
View File
@@ -0,0 +1,402 @@
# Sound and word recognition for Android
This tutorial shows you how to use TensorFlow Lite with pre-built machine
learning models to recognize sounds and spoken words in an Android app. Audio
classification models like the ones shown in this tutorial can be used to detect
activity, identify actions, or recognize voice commands.
![Audio recognition animated demo](https://storage.googleapis.com/download.tensorflow.org/tflite/examples/audio_classification.gif){: .attempt-right}
This tutorial shows you how to download the example code, load the project into
[Android Studio](https://developer.android.com/studio/), and explains key parts
of the code example so you can start adding this functionality to your own app.
The example app code uses the TensorFlow
[Task Library for Audio](https://www.tensorflow.org/lite/inference_with_metadata/task_library/audio_classifier),
which handles most of the audio data recording and preprocessing. For more
information on how audio is pre-processed for use with machine learning models,
see
[Audio Data Preparation and Augmentation](https://www.tensorflow.org/io/tutorials/audio).
## Audio classification with machine learning
The machine learning model in this tutorial recognizes sounds or words from
audio samples recorded with a microphone on an Android device. The example app
in this tutorial allows you to switch between the
[YAMNet/classifier](https://tfhub.dev/google/lite-model/yamnet/classification/tflite/1),
a model that recognizes sounds, and a model that recognizes specific spoken
words, that was
[trained](https://ai.google.dev/edge/litert/libraries/modify/speech_recognition)
using the TensorFlow Lite
[Model Maker](https://www.tensorflow.org/lite/models/modify/model_maker) tool.
The models run predictions on audio clips that contain 15600 individual samples
per clip and are about 1 second in length.
## Setup and run example
For the first part of this tutorial, you download the sample from GitHub and run
it using Android Studio. The following sections of this tutorial explore the
relevant sections of the example, so you can apply them to your own Android
apps.
### System requirements
* [Android Studio](https://developer.android.com/studio/index.html)
version 2021.1.1 (Bumblebee) or higher.
* Android SDK version 31 or higher
* Android device with a minimum OS version of SDK 24 (Android 7.0 -
Nougat) with developer mode enabled.
### Get the example code
Create a local copy of the example code. You will use this code to create a
project in Android Studio and run the sample application.
To clone and setup the example code:
1. Clone the git repository
<pre class="devsite-click-to-copy">
git clone https://github.com/tensorflow/examples.git
</pre>
1. Optionally, configure your git instance to use sparse checkout, so you
have only the files for the example app:
```
cd examples
git sparse-checkout init --cone
git sparse-checkout set lite/examples/audio_classification/android
```
### Import and run the project
Create a project from the downloaded example code, build the project, and then
run it.
To import and build the example code project:
1. Start [Android Studio](https://developer.android.com/studio).
1. In Android Studio, choose **File > New > Import Project**.
1. Navigate to the example code directory containing the `build.gradle` file
(`.../examples/lite/examples/audio_classification/android/build.gradle`)
and select that directory.
If you select the correct directory, Android Studio creates a new project and
builds it. This process can take a few minutes, depending on the speed of your
computer and if you have used Android Studio for other projects. When the build
completes, the Android Studio displays a `BUILD SUCCESSFUL` message in the
**Build Output** status panel.
To run the project:
1. From Android Studio, run the project by selecting **Run > Run 'app'**.
1. Select an attached Android device with a microphone to test the app.
Note: If you use an emulator to run the app, make sure you
[enable audio input](https://developer.android.com/studio/releases/emulator#29.0.6-host-audio)
from the host machine.
The next sections show you the modifications you need to make to your existing
project to add this functionality to your own app, using this example app as a
reference point.
## Add project dependencies
In your own application, you must add specific project dependencies to run
TensorFlow Lite machine learning models, and access utility functions that
convert standard data formats, such as audio, into a tensor data format that can
be processed by the model you are using.
The example app uses the following TensorFlow Lite libraries:
- [TensorFlow Lite Task library Audio API](https://www.tensorflow.org/lite/api_docs/java/org/tensorflow/lite/task/audio/classifier/package-summary) -
Provides the required audio data input classes, execution of the machine
learning model, and output results from the model processing.
The following instructions show how to add the required project dependencies to
your own Android app project.
To add module dependencies:
1. In the module that uses TensorFlow Lite, update the module's
`build.gradle` file to include the following dependencies. In the example
code, this file is located here:
`.../examples/lite/examples/audio_classification/android/build.gradle`
```
dependencies {
...
implementation 'org.tensorflow:tensorflow-lite-task-audio'
}
```
1. In Android Studio, sync the project dependencies by selecting: **File >
Sync Project with Gradle Files**.
## Initialize the ML model
In your Android app, you must initialize the TensorFlow Lite machine learning
model with parameters before running predictions with the model. These
initialization parameters are dependent on the model and can include settings
such as default minimum accuracy thresholds for predictions and labels for words
or sounds that the model can recognize.
A TensorFlow Lite model includes a `*.tflite` file containing the model. The
model file contains the prediction logic and typically includes
[metadata](../../models/convert/metadata.md) about how to interpret prediction
results, such as prediction class names. Model files should be stored in the
`src/main/assets` directory of your development project, as in the code example:
- `<project>/src/main/assets/yamnet.tflite`
For convenience and code readability, the example declares a companion object
that defines the settings for the model.
To initialize the model in your app:
1. Create a companion object to define the settings for the model:
```
companion object {
const val DISPLAY_THRESHOLD = 0.3f
const val DEFAULT_NUM_OF_RESULTS = 2
const val DEFAULT_OVERLAP_VALUE = 0.5f
const val YAMNET_MODEL = "yamnet.tflite"
const val SPEECH_COMMAND_MODEL = "speech.tflite"
}
```
1. Create the settings for the model by building an
`AudioClassifier.AudioClassifierOptions` object:
```
val options = AudioClassifier.AudioClassifierOptions.builder()
.setScoreThreshold(classificationThreshold)
.setMaxResults(numOfResults)
.setBaseOptions(baseOptionsBuilder.build())
.build()
```
1. Use this settings object to construct a TensorFlow Lite
[`AudioClassifier`](https://www.tensorflow.org/lite/inference_with_metadata/task_library/audio_classifier)
object that contains the model:
```
classifier = AudioClassifier.createFromFileAndOptions(context, "yamnet.tflite", options)
```
### Enable hardware acceleration
When initializing a TensorFlow Lite model in your app, you should consider using
hardware acceleration features to speed up the prediction calculations of the
model. TensorFlow Lite
[delegates](https://www.tensorflow.org/lite/performance/delegates) are software
modules that accelerate execution of machine learning models using specialized
processing hardware on a mobile device, such as graphics processing unit (GPUs)
or tensor processing units (TPUs). The code example uses the NNAPI Delegate to
handle hardware acceleration of the model execution:
```
val baseOptionsBuilder = BaseOptions.builder()
.setNumThreads(numThreads)
...
when (currentDelegate) {
DELEGATE_CPU -> {
// Default
}
DELEGATE_NNAPI -> {
baseOptionsBuilder.useNnapi()
}
}
```
Using delegates for running TensorFlow Lite models is recommended, but not
required. For more information about using delegates with TensorFlow Lite, see
[TensorFlow Lite Delegates](https://www.tensorflow.org/lite/performance/delegates).
## Prepare data for the model
In your Android app, your code provides data to the model for interpretation by
transforming existing data such as audio clips into a
[Tensor](https://www.tensorflow.org/lite/api_docs/java/org/tensorflow/lite/Tensor)
data format that can be processed by your model. The data in a Tensor you pass
to a model must have specific dimensions, or shape, that matches the format of
data used to train the model.
The
[YAMNet/classifier model](https://tfhub.dev/google/lite-model/yamnet/classification/tflite/1)
and the customized
[speech commands](https://ai.google.dev/edge/litert/libraries/modify/speech_recognition)
models used in this code example accepts Tensor data objects that represent
single-channel, or mono, audio clips recorded at 16kHz in 0.975 second clips
(15600 samples). Running predictions on new audio data, your app must transform
that audio data into Tensor data objects of that size and shape. The TensorFlow
Lite Task Library
[Audio API](https://www.tensorflow.org/lite/inference_with_metadata/task_library/audio_classifier)
handles the data transformation for you.
In the example code `AudioClassificationHelper` class, the app records live
audio from the device microphones using an Android
[AudioRecord](https://developer.android.com/reference/android/media/AudioRecord)
object. The code uses
[AudioClassifier](https://www.tensorflow.org/lite/api_docs/java/org/tensorflow/lite/task/audio/classifier/AudioClassifier)
to build and configure that object to record audio at a sampling rate
appropriate for the model. The code also uses AudioClassifier to build a
[TensorAudio](https://www.tensorflow.org/lite/api_docs/java/org/tensorflow/lite/support/audio/TensorAudio)
object to store the transformed audio data. Then the TensorAudio object is
passed to the model for analysis.
To provide audio data to the ML model:
- Use the `AudioClassifier` object to create a `TensorAudio` object and a
`AudioRecord` object:
```
fun initClassifier() {
...
try {
classifier = AudioClassifier.createFromFileAndOptions(context, currentModel, options)
// create audio input objects
tensorAudio = classifier.createInputTensorAudio()
recorder = classifier.createAudioRecord()
}
```
Note: Your app must request permission to record audio using an Android device
microphone. See the `fragments/PermissionsFragment` class in the project for an
example. For more information on requesting permissions, see
[Permissions on Android](https://developer.android.com/guide/topics/permissions/overview).
## Run predictions
In your Android app, once you have connected an
[AudioRecord](https://developer.android.com/reference/android/media/AudioRecord)
object and a
[TensorAudio](https://www.tensorflow.org/lite/api_docs/java/org/tensorflow/lite/support/audio/TensorAudio)
object to an AudioClassifier object, you can run the model against that data to
produce a prediction, or *inference*. The example code for this tutorial runs
predictions on clips from a live-recorded audio input stream at a specific
rate.
Model execution consumes significant resources, so it's important to run ML
model predictions on a separate, background thread. The example app uses a
`[ScheduledThreadPoolExecutor](https://developer.android.com/reference/java/util/concurrent/ScheduledThreadPoolExecutor)`
object to isolate the model processing from other functions of the app.
Audio classification models that recognize sounds with a clear beginning and
end, such as words, can produce more accurate predictions on an incoming audio
stream by analyzing overlapping audio clips. This approach
helps the model avoid missing predictions for words that are cut off at the end
of a clip. In the example app, each time you run a prediction the code grabs the
latest 0.975 second clip from the audio recording buffer and analyzes it. You
can make the model analyze overlapping audio clips by setting the model analysis
thread execution pool `interval` value to a length that's shorter than the
length of the clips being analyzed. For example, if your model analyzes 1 second
clips and you set the interval to 500 milliseconds, the model will analyze the
last half of the previous clip and 500 milliseconds of new audio data each time,
creating a clip analysis overlap of 50%.
To start running predictions on the audio data:
1. Use the `AudioClassificationHelper.startAudioClassification()` method
to start the audio recording for the model:
```
fun startAudioClassification() {
if (recorder.recordingState == AudioRecord.RECORDSTATE_RECORDING) {
return
}
recorder.startRecording()
}
```
1. Set how frequently the model generates an inference from the audio
clips by setting a fixed rate `interval` in the
`ScheduledThreadPoolExecutor` object:
```
executor = ScheduledThreadPoolExecutor(1)
executor.scheduleAtFixedRate(
classifyRunnable,
0,
interval,
TimeUnit.MILLISECONDS)
```
1. The `classifyRunnable` object in the code above executes the
`AudioClassificationHelper.classifyAudio()` method, which loads the latest
available audio data from the recorder and performs a prediction:
```
private fun classifyAudio() {
tensorAudio.load(recorder)
val output = classifier.classify(tensorAudio)
...
}
```
Caution: Do not run the ML model predictions on the main execution thread of
your application. Doing so can cause your app user interface to become slow or
unresponsive.
### Stop prediction processing
Make sure your app code stops doing audio classification when your app's audio
processing Fragment or Activity loses focus. Running a machine learning model
continuously has a significant impact on the battery life of an Android device.
Use the `onPause()` method of the Android activity or fragment associated with
the audio classification to stop audio recording and prediction processing.
To stop audio recording and classification:
- Use the `AudioClassificationHelper.stopAudioClassification()` method to
stop recording and the model execution, as shown below in the
`AudioFragment` class:
```
override fun onPause() {
super.onPause()
if (::audioHelper.isInitialized ) {
audioHelper.stopAudioClassification()
}
}
```
## Handle model output
In your Android app, after you process an audio clip, the model produces a list
of predictions which your app code must handle by executing additional business
logic, displaying results to the user, or taking other actions. The output of
any given TensorFlow Lite model varies in terms of the number of predictions it
produces (one or many), and the descriptive information for each prediction. In
the case of the models in the example app, the predictions are either a list of
recognized sounds or words. The AudioClassifier options object used in the code
example lets you set the maximum number of predictions with the
`setMaxResults()` method, as shown in
[Initialize the ML model](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/android/tutorials/audio_classification.md#initialize-the-ml-model)
section.
To get the prediction results from the model:
1. Get the results of the
[AudioClassifier](https://www.tensorflow.org/lite/api_docs/java/org/tensorflow/lite/task/audio/classifier/AudioClassifier)
object's `classify()` method and pass them to the listener object (code
reference):
```
private fun classifyAudio() {
...
val output = classifier.classify(tensorAudio)
listener.onResult(output[0].categories, inferenceTime)
}
```
1. Use the listener's onResult() function handle the output by executing
business logic or displaying results to the user:
```
private val audioClassificationListener = object : AudioClassificationListener {
override fun onResult(results: List<Category>, inferenceTime: Long) {
requireActivity().runOnUiThread {
adapter.categoryList = results
adapter.notifyDataSetChanged()
fragmentAudioBinding.bottomSheetLayout.inferenceTimeVal.text =
String.format("%d ms", inferenceTime)
}
}
```
The model used in this example generates a list of predictions with a label
for the classified sound or word, and a prediction score between 0 and 1 as a
Float representing the confidence of the prediction, with 1 being the highest
confidence rating. In general, predictions with a score below 50% (0.5) are
considered inconclusive. However, how you handle low-value prediction results is
up to you and the needs of your application.
Once the model has returned a set of prediction results, your application can
act on those predictions by presenting the result to your user or executing
additional logic. In the case of the example code, the application lists the
identified sounds or words in the app user interface.
## Next steps
You can find additional TensorFlow Lite models for audio processing on
[TensorFlow Hub](https://tfhub.dev/s?deployment-format=lite&module-type=audio-embedding,audio-pitch-extraction,audio-event-classification,audio-stt)
and through the
[Pre-trained models guide](https://www.tensorflow.org/lite/models/trained) page.
For more information about implementing machine learning in your mobile
application with TensorFlow Lite, see the
[TensorFlow Lite Developer Guide](https://www.tensorflow.org/lite/guide).
tensorflow/lite/g3doc/android/tutorials/object_detection.md
+491
View File
@@ -0,0 +1,491 @@
# Object detection with Android
This tutorial shows you how to build an Android app using TensorFlow Lite to
continuously detect objects in frames captured by a device camera. This
application is designed for a physical Android device. If you are updating an
existing project, you can use the code sample as a reference and skip ahead to
the instructions for [modifying your project](#add_dependencies).
![Object detection animated demo](https://storage.googleapis.com/download.tensorflow.org/tflite/examples/obj_detection_cat.gif){: .attempt-right width="250px"}
## Object detection overview
*Object detection* is the machine learning task of identifying the presence and
location of multiple classes of objects within an image. An object detection
model is trained on a dataset that contains a set of known objects.
The trained model receives image frames as input and attempts to categorize
items in the images from the set of known classes it was trained to recognize.
For each image frame, the object detection model outputs a list of the objects
it detects, the location of a bounding box for each object, and a score that
indicates the confidence of the object being correctly classified.
## Models and dataset
This tutorial uses models that were trained using the
[COCO dataset](http://cocodataset.org/). COCO is a large-scale object detection
dataset that contains 330K images, 1.5 million object instances, and 80 object
categories.
You have the option to use one of the following pre-trained models:
* [EfficientDet-Lite0](https://tfhub.dev/tensorflow/lite-model/efficientdet/lite0/detection/metadata/1)
*[Recommended]* - a lightweight object detection model with a BiFPN feature
extractor, shared box predictor, and focal loss. The mAP (mean Average
Precision) for the COCO 2017 validation dataset is 25.69%.
* [EfficientDet-Lite1](https://tfhub.dev/tensorflow/lite-model/efficientdet/lite1/detection/metadata/1) -
a medium-sized EfficientDet object detection model. The mAP for the COCO
2017 validation dataset is 30.55%.
* [EfficientDet-Lite2](https://tfhub.dev/tensorflow/lite-model/efficientdet/lite2/detection/metadata/1) -
a larger EfficientDet object detection model. The mAP for the COCO 2017
validation dataset is 33.97%.
* [MobileNetV1-SSD](https://tfhub.dev/tensorflow/lite-model/ssd_mobilenet_v1/1/metadata/2) -
an extremely lightweight model optimized to work with TensorFlow Lite for
object detection. The mAP for the COCO 2017 validation dataset is 21%.
For this tutorial, the *EfficientDet-Lite0* model strikes a good balance between
size and accuracy.
Downloading, extraction, and placing the models into the assets folder is
managed automatically by the `download.gradle` file, which is run at build time.
You don't need to manually download TFLite models into the project.
## Setup and run example
To setup the object detection app, download the sample from
[GitHub](https://github.com/tensorflow/examples/tree/master/lite/examples/object_detection/android)
and run it using [Android Studio](https://developer.android.com/studio/). The
following sections of this tutorial explore the relevant sections of the code
example, so you can apply them to your own Android apps.
### System requirements
* **[Android Studio](https://developer.android.com/studio/index.html)**
version 2021.1.1 (Bumblebee) or higher.
* Android SDK version 31 or higher
* Android device with a minimum OS version of SDK 24 (Android 7.0 -
Nougat) with developer mode enabled.
Note: This example uses a camera, so run it on a physical Android device.
### Get the example code
Create a local copy of the example code. You will use this code to create a
project in Android Studio and run the sample application.
To clone and setup the example code:
1. Clone the git repository
<pre class="devsite-click-to-copy">
git clone https://github.com/tensorflow/examples.git
</pre>
2. Optionally, configure your git instance to use sparse checkout, so you have only
the files for the object detection example app:
<pre class="devsite-click-to-copy">
cd examples
git sparse-checkout init --cone
git sparse-checkout set lite/examples/object_detection/android
</pre>
### Import and run the project
Create a project from the downloaded example code, build the project, and then
run it.
To import and build the example code project:
1. Start [Android Studio](https://developer.android.com/studio).
1. From the Android Studio, select **File > New > Import Project**.
1. Navigate to the example code directory containing the build.gradle file
(`.../examples/lite/examples/object_detection/android/build.gradle`) and
select that directory.
1. If Android Studio requests a Gradle Sync, choose OK.
1. Ensure that your Android device is connected to your computer and developer
mode is enabled. Click the green `Run` arrow.
If you select the correct directory, Android Studio creates a new project and
builds it. This process can take a few minutes, depending on the speed of your
computer and if you have used Android Studio for other projects. When the build
completes, the Android Studio displays a `BUILD SUCCESSFUL` message in the
**Build Output** status panel.
Note: The example code is built with Android Studio 4.2.2, but works with
earlier versions of Studio. If you are using an earlier version of Android
Studio you can try adjust the version number of the Android plugin so that the
build completes, instead of upgrading Studio.
**Optional:** To fix build errors by updating the Android plugin version:
1. Open the build.gradle file in the project directory.
1. Change the Android tools version as follows:
```
// from: classpath
'com.android.tools.build:gradle:4.2.2'
// to: classpath
'com.android.tools.build:gradle:4.1.2'
```
1. Sync the project by selecting: **File > Sync Project with Gradle Files**.
To run the project:
1. From Android Studio, run the project by selecting **Run > Run…**.
1. Select an attached Android device with a camera to test the app.
The next sections show you the modifications you need to make to your existing
project to add this functionality to your own app, using this example app as a
reference point.
## Add project dependencies {:#add_dependencies}
In your own application, you must add specific project dependencies to run
TensorFlow Lite machine learning models, and access utility functions that
convert data such as images, into a tensor data format that can be processed by
the model you are using.
The example app uses the TensorFlow Lite
[Task library for vision](../../inference_with_metadata/task_library/overview.md#supported-tasks)
to enable execution of the object detection machine learning model. The
following instructions explain how to add the required library dependencies to
your own Android app project.
The following instructions explain how to add the required project and module
dependencies to your own Android app project.
To add module dependencies:
1. In the module that uses TensorFlow Lite, update the module's `build.gradle`
file to include the following dependencies. In the example code, this file
is located here:
`...examples/lite/examples/object_detection/android/app/build.gradle`
([code reference](https://github.com/tensorflow/examples/blob/master/lite/examples/object_detection/android/app/build.gradle))
```
dependencies {
...
implementation 'org.tensorflow:tensorflow-lite-task-vision:0.4.0'
// Import the GPU delegate plugin Library for GPU inference
implementation 'org.tensorflow:tensorflow-lite-gpu-delegate-plugin:0.4.0'
implementation 'org.tensorflow:tensorflow-lite-gpu:2.9.0'
}
```
The project must include the Vision task library
(`tensorflow-lite-task-vision`). The graphics processing unit (GPU) library
(`tensorflow-lite-gpu-delegate-plugin`) provides the infrastructure to run
the app on GPU, and Delegate (`tensorflow-lite-gpu`) provides the
compatibility list.
1. In Android Studio, sync the project dependencies by selecting: **File > Sync
Project with Gradle Files**.
## Initialize the ML model
In your Android app, you must initialize the TensorFlow Lite machine learning
model with parameters before running predictions with the model. These
initialization parameters are consistent across object detection models and can
include settings such as minimum accuracy thresholds for predictions.
A TensorFlow Lite model includes a `.tflite` file containing the model code and
frequently includes a labels file containing the names of the classes predicted
by the model. In the case of object detection, classes are objects such as a
person, dog, cat, or car.
This example downloads several models that are specified in
`download_models.gradle`, and the `ObjectDetectorHelper` class provides a
selector for the models:
```
val modelName =
when (currentModel) {
MODEL_MOBILENETV1 -> "mobilenetv1.tflite"
MODEL_EFFICIENTDETV0 -> "efficientdet-lite0.tflite"
MODEL_EFFICIENTDETV1 -> "efficientdet-lite1.tflite"
MODEL_EFFICIENTDETV2 -> "efficientdet-lite2.tflite"
else -> "mobilenetv1.tflite"
}
```
Key Point: Models should be stored in the `src/main/assets` directory of your
development project. The TensorFlow Lite Task library automatically checks this
directory when you specify a model file name.
To initialize the model in your app:
1. Add a `.tflite` model file to the `src/main/assets` directory of your
development project, such as:
[EfficientDet-Lite0](https://tfhub.dev/tensorflow/lite-model/efficientdet/lite0/detection/metadata/1).
1. Set a static variable for your model's file name. In the example app, you
set the `modelName` variable to `MODEL_EFFICIENTDETV0` to use the
EfficientDet-Lite0 detection model.
1. Set the options for model, such as the prediction threshold, results set
size, and optionally, hardware acceleration delegates:
```
val optionsBuilder =
ObjectDetector.ObjectDetectorOptions.builder()
.setScoreThreshold(threshold)
.setMaxResults(maxResults)
```
1. Use the settings from this object to construct a TensorFlow Lite
[`ObjectDetector`](https://www.tensorflow.org/lite/api_docs/java/org/tensorflow/lite/task/vision/detector/ObjectDetector#createFromFile\(Context,%20java.lang.String\))
object that contains the model:
```
objectDetector =
ObjectDetector.createFromFileAndOptions(
context, modelName, optionsBuilder.build())
```
The `setupObjectDetector` sets up the following model parameters:
* Detection threshold
* Maximum number of detection results
* Number of processing threads to use
(`BaseOptions.builder().setNumThreads(numThreads)`)
* Actual model (`modelName`)
* ObjectDetector object (`objectDetector`)
### Configure hardware accelerator
When initializing a TensorFlow Lite model in your application, you can use
hardware acceleration features to speed up the prediction calculations of the
model.
TensorFlow Lite *delegates* are software modules that accelerate execution of
machine learning models using specialized processing hardware on a mobile
device, such as Graphics Processing Units (GPUs), Tensor Processing Units
(TPUs), and Digital Signal Processors (DSPs). Using delegates for running
TensorFlow Lite models is recommended, but not required.
The object detector is initialized using the current settings on the thread that
is using it. You can use CPU and [NNAPI](../../android/delegates/nnapi.md)
delegates with detectors that are created on the main thread and used on a
background thread, but the thread that initialized the detector must use the GPU
delegate.
The delegates are set within the `ObjectDetectionHelper.setupObjectDetector()`
function:
```
when (currentDelegate) {
DELEGATE_CPU -> {
// Default
}
DELEGATE_GPU -> {
if (CompatibilityList().isDelegateSupportedOnThisDevice) {
baseOptionsBuilder.useGpu()
} else {
objectDetectorListener?.onError("GPU is not supported on this device")
}
}
DELEGATE_NNAPI -> {
baseOptionsBuilder.useNnapi()
}
}
```
For more information about using hardware acceleration delegates with TensorFlow
Lite, see [TensorFlow Lite Delegates](../../performance/delegates.md).
## Prepare data for the model
In your Android app, your code provides data to the model for interpretation by
transforming existing data such as image frames into a Tensor data format that
can be processed by your model. The data in a Tensor you pass to a model must
have specific dimensions, or shape, that matches the format of data used to
train the model.
The
[EfficientDet-Lite0](https://tfhub.dev/tensorflow/lite-model/efficientdet/lite0/detection/metadata/1)
model used in this code example accepts Tensors representing images with a
dimension of 320 x 320, with three channels (red, blue, and green) per pixel.
Each value in the tensor is a single byte between 0 and 255. So, to run
predictions on new images, your app must transform that image data into Tensor
data objects of that size and shape. The TensorFlow Lite Task Library Vision API
handles the data transformation for you.
The app uses an
[`ImageAnalysis`](https://developer.android.com/training/camerax/analyze) object
to pull images from the camera. This object calls the `detectObject` function
with bitmap from the camera. The data is automatically resized and rotated by
`ImageProcessor` so that it meets the image data requirements of the model. The
image is then translated into a
[`TensorImage`](https://www.tensorflow.org/lite/api_docs/java/org/tensorflow/lite/support/image/TensorImage)
object.
To prepare data from the camera subsystem to be processed by the ML model:
1. Build an `ImageAnalysis` object to extract images in the required format:
```
imageAnalyzer =
ImageAnalysis.Builder()
.setTargetAspectRatio(AspectRatio.RATIO_4_3)
.setTargetRotation(fragmentCameraBinding.viewFinder.display.rotation)
.setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
.setOutputImageFormat(OUTPUT_IMAGE_FORMAT_RGBA_8888)
.build()
...
```
1. Connect the analyzer to the camera subsystem and create a bitmap buffer to
contain the data received from the camera:
```
.also {
it.setAnalyzer(cameraExecutor) {
image -> if (!::bitmapBuffer.isInitialized)
{ bitmapBuffer = Bitmap.createBitmap( image.width, image.height,
Bitmap.Config.ARGB_8888 ) } detectObjects(image)
}
}
```
1. Extract the specific image data needed by the model, and pass the image
rotation information:
```
private fun detectObjects(image: ImageProxy) {
//Copy out RGB bits to the shared bitmap buffer
image.use {bitmapBuffer.copyPixelsFromBuffer(image.planes[0].buffer) }
val imageRotation = image.imageInfo.rotationDegrees
objectDetectorHelper.detect(bitmapBuffer, imageRotation)
}
```
1. Complete any final data transformations and add the image data to a
`TensorImage` object, as shown in the `ObjectDetectorHelper.detect()` method
of the example app:
```
val imageProcessor = ImageProcessor.Builder().add(Rot90Op(-imageRotation / 90)).build()
// Preprocess the image and convert it into a TensorImage for detection.
val tensorImage = imageProcessor.process(TensorImage.fromBitmap(image))
```
Note: When extracting image information from the Android camera subsystem, make
sure to get an image in RGB format. This format is required by the TensorFlow
Lite ImageProcessor class which you use to prepare the image for analysis by a
model. If the RGB-format image contains an Alpha channel, that transparency data
is ignored.
## Run predictions
In your Android app, once you create a TensorImage object with image data in the
correct format, you can run the model against that data to produce a prediction,
or *inference*.
In the `fragments/CameraFragment.kt` class of the example app, the
`imageAnalyzer` object within the `bindCameraUseCases` function automatically
passes data to the model for predictions when the app is connected to the
camera.
The app uses the `cameraProvider.bindToLifecycle()` method to handle the camera
selector, preview window, and ML model processing. The `ObjectDetectorHelper.kt`
class handles passing the image data into the model. To run the model and
generate predictions from image data:
- Run the prediction by passing the image data to your predict function:
```
val results = objectDetector?.detect(tensorImage)
```
The TensorFlow Lite Interpreter object receives this data, runs it against the
model, and produces a list of predictions. For continuous processing of data by
the model, use the `runForMultipleInputsOutputs()` method so that Interpreter
objects are not created and then removed by the system for each prediction run.
## Handle model output
In your Android app, after you run image data against the object detection
model, it produces a list of predictions which your app code must handle by
executing additional business logic, displaying results to the user, or taking
other actions.
The output of any given TensorFlow Lite model varies in terms of the number of
predictions it produces (one or many), and the descriptive information for each
prediction. In the case of an object detection model, predictions typically
include data for a bounding box that indicates where an object is detected in
the image. In the example code, the results are passed to the `onResults`
function in `CameraFragment.kt`, which is acting as a DetectorListener on the
object detection process.
```
interface DetectorListener {
fun onError(error: String)
fun onResults(
results: MutableList<Detection>?,
inferenceTime: Long,
imageHeight: Int,
imageWidth: Int
)
}
```
For the model used in this example, each prediction includes a bounding box
location for the object, a label for the object, and a prediction score between
0 and 1 as a Float representing the confidence of the prediction, with 1 being
the highest confidence rating. In general, predictions with a score below 50%
(0.5) are considered inconclusive. However, how you handle low-value prediction
results is up to you and the needs of your application.
To handle model prediction results:
1. Use a listener pattern to pass results to your app code or user interface
objects. The example app uses this pattern to pass detection results from
the `ObjectDetectorHelper` object to the `CameraFragment` object:
```
objectDetectorListener.onResults(
// instance of CameraFragment
results,
inferenceTime,
tensorImage.height,
tensorImage.width)
```
1. Act on the results, such as displaying the prediction to the user. The
example draws an overlay on the CameraPreview object to show the result:
```
override fun onResults(
results: MutableList<Detection>?,
inferenceTime: Long,
imageHeight: Int,
imageWidth: Int
) {
activity?.runOnUiThread {
fragmentCameraBinding.bottomSheetLayout.inferenceTimeVal.text =
String.format("%d ms", inferenceTime)
// Pass necessary information to OverlayView for drawing on the canvas
fragmentCameraBinding.overlay.setResults(
results ?: LinkedList<Detection>(),
imageHeight,
imageWidth
)
// Force a redraw
fragmentCameraBinding.overlay.invalidate()
}
}
```
Once the model has returned a prediction result, your application can act on
that prediction by presenting the result to your user or executing additional
logic. In the case of the example code, the application draws a bounding box
around the identified object and displays the class name on screen.
## Next steps
* Explore various uses of TensorFlow Lite in the [examples](../../examples).
* Learn more about using machine learning models with TensorFlow Lite in the
[Models](../../models) section.
* Learn more about implementing machine learning in your mobile application in
the [TensorFlow Lite Developer Guide](../../guide).
tensorflow/lite/g3doc/android/tutorials/question_answer.md
+534
View File
@@ -0,0 +1,534 @@
# Answering questions with Android
![Question answering example app in Android](../../examples/bert_qa/images/screenshot.gif){: .attempt-right width="250px"}
This tutorial shows you how to build an Android application using TensorFlow
Lite to provide answers to questions structured in natural language text. The
[example application](https://github.com/tensorflow/examples/tree/master/lite/examples/bert_qa/android)
uses the *BERT question answerer*
([`BertQuestionAnswerer`](../../inference_with_metadata/task_library/bert_question_answerer))
API within the
[Task library for natural language (NL)](https://ai.google.dev/edge/litert/libraries/task_library/overview)
to enable Question answering machine learning models. The application is
designed for a physical Android device but can also run on a device emulator.
If you are updating an existing project, you can use the example application as
a reference or template. For instructions on how to add question answering to an
existing application, refer to
[Updating and modifying your application](#modify_applications).
## Question answering overview
*Question answering* is the machine learning task of answering questions posed
in natural language. A trained question answering model receives a text passage
and question as input, and attempts to answer the question based on its
interpretation of the information within the passage.
A Question answering model is trained on a question answering dataset, which
consists of a reading comprehension dataset along with question-answer pairs
based on different segments of text.
For more information on how the models in this tutorial are generated, refer to
the
[BERT Question Answer with TensorFlow Lite Model Maker](https://ai.google.dev/edge/litert/libraries/modify/question_answer)
tutorial.
## Models and dataset
The example app uses the Mobile BERT Q&A
([`mobilebert`](https://tfhub.dev/tensorflow/lite-model/mobilebert/1/metadata/1))
model, which is a lighter and faster version of
[BERT](https://arxiv.org/abs/1810.04805) (Bidirectional Encoder Representations
from Transformers). For more information on `mobilebert`, see the
[MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices](https://arxiv.org/abs/2004.02984)
research paper.
The `mobilebert` model was trained using the Stanford Question Answering Dataset
([SQuAD](https://rajpurkar.github.io/SQuAD-explorer/)) dataset, a reading
comprehension dataset consisting of articles from Wikipedia and a set of
question-answer pairs for each article.
## Setup and run the example app
To setup the question answering application, download the example app from
[GitHub](https://github.com/tensorflow/examples/tree/master/lite/examples/bert_qa/android)
and run it using [Android Studio](https://developer.android.com/studio/).
### System requirements
* **[Android Studio](https://developer.android.com/studio/index.html)**
version 2021.1.1 (Bumblebee) or higher.
* Android SDK version 31 or higher
* Android device with a minimum OS version of SDK 21 (Android 7.0 - Nougat)
with
[developer mode](https://developer.android.com/studio/debug/dev-options)
enabled, or an Android Emulator.
### Get the example code
Create a local copy of the example code. You will use this code to create a
project in Android Studio and run the example application.
To clone and setup the example code:
1. Clone the git repository
<pre class="devsite-click-to-copy">
git clone https://github.com/tensorflow/examples.git
</pre>
2. Optionally, configure your git instance to use sparse checkout, so you have only
the files for the question answering example app:
<pre class="devsite-click-to-copy">
cd examples
git sparse-checkout init --cone
git sparse-checkout set lite/examples/bert_qa/android
</pre>
### Import and run the project
Create a project from the downloaded example code, build the project, and then
run it.
To import and build the example code project:
1. Start [Android Studio](https://developer.android.com/studio).
1. From the Android Studio, select **File > New > Import Project**.
1. Navigate to the example code directory containing the build.gradle file
(`.../examples/lite/examples/bert_qa/android/build.gradle`) and select that
directory.
1. If Android Studio requests a Gradle Sync, choose OK.
1. Ensure that your Android device is connected to your computer and developer
mode is enabled. Click the green `Run` arrow.
If you select the correct directory, Android Studio creates a new project and
builds it. This process can take a few minutes, depending on the speed of your
computer and if you have used Android Studio for other projects. When the build
completes, the Android Studio displays a `BUILD SUCCESSFUL` message in the
**Build Output** status panel.
To run the project:
1. From Android Studio, run the project by selecting **Run > Run…**.
1. Select an attached Android device (or emulator) to test the app.
### Using the application
After running the project in Android Studio, the application automatically opens
on the connected device or device emulator.
To use the Question answerer example app:
1. Choose a topic from the list of subjects.
1. Choose a suggested question or enter your own in the text box.
1. Toggle the orange arrow to run the model.
The application attempts to identify the answer to the question from the passage
text. If the model detects an answer within the passage, the application
highlights the relevant span of text for the user.
You now have a functioning question answering application. Use the following
sections to better understand how the example application works, and how to
implement question answering features in your production applications:
* [How the application works](#how_it_works) - A walkthrough of the structure
and key files of the example application.
* [Modify your application](#modify_applications) - Instructions on adding
question answering to an existing application.
## How the example app works {:#how_it_works}
The application uses the `BertQuestionAnswerer` API within the
[Task library for natural language (NL)](https://ai.google.dev/edge/litert/libraries/task_library/overview)
package. The MobileBERT model was trained using the TensorFlow Lite
[Model Maker](https://ai.google.dev/edge/litert/libraries/modify/question_answer).
The application runs on CPU by default, with the option of hardware acceleration
using the GPU or NNAPI delegate.
The following files and directories contain the crucial code for this
application:
* [BertQaHelper.kt](https://github.com/tensorflow/examples/blob/master/lite/examples/bert_qa/android/app/src/main/java/org/tensorflow/lite/examples/bertqa/BertQaHelper.kt) -
Initializes the question answerer and handles the model and delegate
selection.
* [QaFragment.kt](https://github.com/tensorflow/examples/blob/master/lite/examples/bert_qa/android/app/src/main/java/org/tensorflow/lite/examples/bertqa/fragments/QaFragment.kt) -
Handles and formats the results.
* [MainActivity.kt](https://github.com/tensorflow/examples/blob/master/lite/examples/bert_qa/android/app/src/main/java/org/tensorflow/lite/examples/bertqa/MainActivity.kt) -
Provides the organizing logic of the app.
## Modify your application {:#modify_applications}
The following sections explain the key steps to modify your own Android app to
run the model shown in the example app. These instructions use the example app
as a reference point. The specific changes needed for your own app may vary from
the example app.
### Open or create an Android project
You need an Android development project in Android Studio to follow along with
the rest of these instructions. Follow the instructions below to open an
existing project or create a new one.
To open an existing Android development project:
* In Android Studio, select *File > Open* and select an existing project.
To create a basic Android development project:
* Follow the instructions in Android Studio to
[Create a basic project](https://developer.android.com/studio/projects/create-project).
For more information on using Android Studio, refer to the
[Android Studio documentation](https://developer.android.com/studio/intro).
### Add project dependencies
In your own application, add specific project dependencies to run TensorFlow
Lite machine learning models and access utility functions. These functions
convert data such as strings into a tensor data format that can be processed by
the model. The following instructions explain how to add the required project
and module dependencies to your own Android app project.
To add module dependencies:
1. In the module that uses TensorFlow Lite, update the module's `build.gradle`
file to include the following dependencies.
In the example application, the dependencies are located in
[app/build.gradle](https://github.com/tensorflow/examples/blob/master/lite/examples/bert_qa/android/app/build.gradle):
```
dependencies {
...
// Import tensorflow library
implementation 'org.tensorflow:tensorflow-lite-task-text:0.3.0'
// Import the GPU delegate plugin Library for GPU inference
implementation 'org.tensorflow:tensorflow-lite-gpu-delegate-plugin:0.4.0'
implementation 'org.tensorflow:tensorflow-lite-gpu:2.9.0'
}
```
The project must include the Text task library
(`tensorflow-lite-task-text`).
If you want to modify this app to run on a graphics processing unit (GPU),
the GPU library (`tensorflow-lite-gpu-delegate-plugin`) provides the
infrastructure to run the app on GPU, and Delegate (`tensorflow-lite-gpu`)
provides the compatibility list.
1. In Android Studio, sync the project dependencies by selecting: **File > Sync
Project with Gradle Files**.
### Initialize the ML models {:#initialize_models}
In your Android app, you must initialize the TensorFlow Lite machine learning
model with parameters before running predictions with the model.
A TensorFlow Lite model is stored as a `*.tflite` file. The model file contains
the prediction logic and typically includes
[metadata](https://ai.google.dev/edge/litert/models/metadata) about how to
interpret prediction results. Typically, model files are stored in the
`src/main/assets` directory of your development project, as in the code example:
- `<project>/src/main/assets/mobilebert_qa.tflite`
Note: The example app uses a
[`download_model.gradle`](https://github.com/tensorflow/examples/blob/master/lite/examples/bert_qa/android/app/download_models.gradle)
file to download the
[mobilebert_qa](https://www.tensorflow.org/lite/inference_with_metadata/task_library/bert_question_answerer)
model and
[passage text](https://storage.googleapis.com/download.tensorflow.org/models/tflite/bert_qa/contents_from_squad.json)
at build time. This approach is not required for a production app.
For convenience and code readability, the example declares a companion object
that defines the settings for the model.
To initialize the model in your app:
1. Create a companion object to define the settings for the model. In the
example application, this object is located in
[BertQaHelper.kt](https://github.com/tensorflow/examples/blob/master/lite/examples/bert_qa/android/app/src/main/java/org/tensorflow/lite/examples/bertqa/BertQaHelper.kt#L100-L106):
```
companion object {
private const val BERT_QA_MODEL = "mobilebert.tflite"
private const val TAG = "BertQaHelper"
const val DELEGATE_CPU = 0
const val DELEGATE_GPU = 1
const val DELEGATE_NNAPI = 2
}
```
1. Create the settings for the model by building a `BertQaHelper` object, and
construct a TensorFlow Lite object with `bertQuestionAnswerer`.
In the example application, this is located in the
`setupBertQuestionAnswerer()` function within
[BertQaHelper.kt](https://github.com/tensorflow/examples/blob/master/lite/examples/bert_qa/android/app/src/main/java/org/tensorflow/lite/examples/bertqa/BertQaHelper.kt#L41-L76):
```
class BertQaHelper(
...
) {
...
init {
setupBertQuestionAnswerer()
}
fun clearBertQuestionAnswerer() {
bertQuestionAnswerer = null
}
private fun setupBertQuestionAnswerer() {
val baseOptionsBuilder = BaseOptions.builder().setNumThreads(numThreads)
...
val options = BertQuestionAnswererOptions.builder()
.setBaseOptions(baseOptionsBuilder.build())
.build()
try {
bertQuestionAnswerer =
BertQuestionAnswerer.createFromFileAndOptions(context, BERT_QA_MODEL, options)
} catch (e: IllegalStateException) {
answererListener
?.onError("Bert Question Answerer failed to initialize. See error logs for details")
Log.e(TAG, "TFLite failed to load model with error: " + e.message)
}
}
...
}
```
### Enable hardware acceleration (optional) {:#hardware_acceleration}
When initializing a TensorFlow Lite model in your app, you should consider using
hardware acceleration features to speed up the prediction calculations of the
model. TensorFlow Lite
[delegates](https://www.tensorflow.org/lite/performance/delegates) are software
modules that accelerate the execution of machine learning models using
specialized processing hardware on a mobile device, such as graphics processing
unit (GPUs) or tensor processing units (TPUs).
To enable hardware acceleration in your app:
1. Create a variable to define the delegate that the application will use. In
the example application, this variable is located early in
[BertQaHelper.kt](https://github.com/tensorflow/examples/blob/master/lite/examples/bert_qa/android/app/src/main/java/org/tensorflow/lite/examples/bertqa/BertQaHelper.kt#L31):
```
var currentDelegate: Int = 0
```
1. Create a delegate selector. In the example application, the delegate
selector is located in the `setupBertQuestionAnswerer` function within
[BertQaHelper.kt](https://github.com/tensorflow/examples/blob/master/lite/examples/bert_qa/android/app/src/main/java/org/tensorflow/lite/examples/bertqa/BertQaHelper.kt#L48-L62):
```
when (currentDelegate) {
DELEGATE_CPU -> {
// Default
}
DELEGATE_GPU -> {
if (CompatibilityList().isDelegateSupportedOnThisDevice) {
baseOptionsBuilder.useGpu()
} else {
answererListener?.onError("GPU is not supported on this device")
}
}
DELEGATE_NNAPI -> {
baseOptionsBuilder.useNnapi()
}
}
```
Using delegates for running TensorFlow Lite models is recommended, but not
required. For more information about using delegates with TensorFlow Lite, see
[TensorFlow Lite Delegates](https://www.tensorflow.org/lite/performance/delegates).
### Prepare data for the model
In your Android app, your code provides data to the model for interpretation by
transforming existing data such as raw text into a
[Tensor](https://www.tensorflow.org/lite/api_docs/java/org/tensorflow/lite/Tensor)
data format that can be processed by your model. The Tensor you pass to a model
must have specific dimensions, or shape, that matches the format of data used to
train the model. This question answering app accepts
[strings](https://developer.android.com/reference/java/lang/String.html) as
inputs for both the text passage and question. The model does not recognize
special characters and non-English words.
To provide passage text data to the model:
1. Use the `LoadDataSetClient` object to load the passage text data to the app.
In the example application, this is located in
[LoadDataSetClient.kt](https://github.com/tensorflow/examples/blob/master/lite/examples/bert_qa/android/app/src/main/java/org/tensorflow/lite/examples/bertqa/dataset/LoadDataSetClient.kt#L25-L45)
```
fun loadJson(): DataSet? {
var dataSet: DataSet? = null
try {
val inputStream: InputStream = context.assets.open(JSON_DIR)
val bufferReader = inputStream.bufferedReader()
val stringJson: String = bufferReader.use { it.readText() }
val datasetType = object : TypeToken<DataSet>() {}.type
dataSet = Gson().fromJson(stringJson, datasetType)
} catch (e: IOException) {
Log.e(TAG, e.message.toString())
}
return dataSet
}
```
1. Use the `DatasetFragment` object to list the titles for each passage of text
and start the **TFL Question and Answer** screen. In the example
application, this is located in
[DatasetFragment.kt](https://github.com/tensorflow/examples/blob/master/lite/examples/bert_qa/android/app/src/main/java/org/tensorflow/lite/examples/bertqa/fragments/DatasetFragment.kt):
```
class DatasetFragment : Fragment() {
...
override fun onViewCreated(view: View, savedInstanceState: Bundle?) {
super.onViewCreated(view, savedInstanceState)
val client = LoadDataSetClient(requireActivity())
client.loadJson()?.let {
titles = it.getTitles()
}
...
}
...
}
```
1. Use the `onCreateViewHolder` function within the `DatasetAdapter` object to
present the titles for each passage of text. In the example application,
this is located in
[DatasetAdapter.kt](https://github.com/tensorflow/examples/blob/master/lite/examples/bert_qa/android/app/src/main/java/org/tensorflow/lite/examples/bertqa/fragments/DatasetAdapter.kt):
```
override fun onCreateViewHolder(parent: ViewGroup, viewType: Int): ViewHolder {
val binding = ItemDatasetBinding.inflate(
LayoutInflater.from(parent.context),
parent,
false
)
return ViewHolder(binding)
}
```
To provide user questions to the model:
1. Use the `QaAdapter` object to provide the question to the model. In the
example application, this is located in
[QaAdapter.kt](https://github.com/tensorflow/examples/blob/master/lite/examples/bert_qa/android/app/src/main/java/org/tensorflow/lite/examples/bertqa/fragments/QaAdapter.kt):
```
class QaAdapter(private val question: List<String>, private val select: (Int) -> Unit) :
RecyclerView.Adapter<QaAdapter.ViewHolder>() {
inner class ViewHolder(private val binding: ItemQuestionBinding) :
RecyclerView.ViewHolder(binding.root) {
init {
binding.tvQuestionSuggestion.setOnClickListener {
select.invoke(adapterPosition)
}
}
fun bind(question: String) {
binding.tvQuestionSuggestion.text = question
}
}
...
}
```
### Run predictions
In your Android app, once you have initialized a
[BertQuestionAnswerer](https://www.tensorflow.org/lite/api_docs/java/org/tensorflow/lite/task/text/qa/BertQuestionAnswerer)
object, you can begin inputting questions in the form of natural language text
to the model. The model attempts to identify the answer within the text passage.
To run predictions:
1. Create an `answer` function, which runs the model and measures the time
taken to identify the answer (`inferenceTime`). In the example application,
the `answer` function is located in
[BertQaHelper.kt](https://github.com/tensorflow/examples/blob/master/lite/examples/bert_qa/android/app/src/main/java/org/tensorflow/lite/examples/bertqa/BertQaHelper.kt#L78-L98):
```
fun answer(contextOfQuestion: String, question: String) {
if (bertQuestionAnswerer == null) {
setupBertQuestionAnswerer()
}
var inferenceTime = SystemClock.uptimeMillis()
val answers = bertQuestionAnswerer?.answer(contextOfQuestion, question)
inferenceTime = SystemClock.uptimeMillis() - inferenceTime
answererListener?.onResults(answers, inferenceTime)
}
```
1. Pass the results from `answer` to the listener object.
```
interface AnswererListener {
fun onError(error: String)
fun onResults(
results: List<QaAnswer>?,
inferenceTime: Long
)
}
```
### Handle model output
After you input a question, the model provides a maximum of five possible
answers within the passage.
To get the results from the model:
1. Create an `onResult` function for the listener object to handle the output.
In the example application, the listener object is located in
[BertQaHelper.kt](https://github.com/tensorflow/examples/blob/master/lite/examples/bert_qa/android/app/src/main/java/org/tensorflow/lite/examples/bertqa/BertQaHelper.kt#L92-L98)
```
interface AnswererListener {
fun onError(error: String)
fun onResults(
results: List<QaAnswer>?,
inferenceTime: Long
)
}
```
1. Highlight sections of the passage based on the results. In the example
application, this is located in
[QaFragment.kt](https://github.com/tensorflow/examples/blob/master/lite/examples/bert_qa/android/app/src/main/java/org/tensorflow/lite/examples/bertqa/fragments/QaFragment.kt#L199-L208):
```
override fun onResults(results: List<QaAnswer>?, inferenceTime: Long) {
results?.first()?.let {
highlightAnswer(it.text)
}
fragmentQaBinding.tvInferenceTime.text = String.format(
requireActivity().getString(R.string.bottom_view_inference_time),
inferenceTime
)
}
```
Once the model has returned a set of results, your application can act on those
predictions by presenting the result to your user or executing additional logic.
## Next steps
* Train and implement the models from scratch with the
[Question Answer with TensorFlow Lite Model Maker](https://ai.google.dev/edge/litert/libraries/modify/question_answer)
tutorial.
* Explore more
[text processing tools for TensorFlow](https://www.tensorflow.org/text).
* Download other BERT models on
[TensorFlow Hub](https://tfhub.dev/google/collections/bert/1).
* Explore various uses of TensorFlow Lite in the [examples](../../examples).
tensorflow/lite/g3doc/android/tutorials/text_classification.md
+488
View File
@@ -0,0 +1,488 @@
# Text classification with Android
This tutorial shows you how to build an Android application using TensorFlow
Lite to classify natural language text. This application is designed for a
physical Android device but can also run on a device emulator.
The
[example application](https://github.com/tensorflow/examples/tree/master/lite/examples/text_classification/android)
uses TensorFlow Lite to classify text as either positive or negative, using the
[Task library for natural language (NL)](https://ai.google.dev/edge/litert/libraries/task_library/overview)
to enable execution of the text classification machine learning models.
If you are updating an existing project, you can use the example application as
a reference or template. For instructions on how to add text classification to
an existing application, refer to
[Updating and modifying your application](#modify_applications).
## Text classification overview
*Text classification* is the machine learning task of assigning a set of
predefined categories to open-ended text. A text classification model is trained
on a corpus of natural language text, where words or phrases are manually
classified.
The trained model receives text as input and attempts to categorize the text
according to the set of known classes it was trained to classify. For example,
the models in this example accept a snippet of text and determines whether the
sentiment of the text is positive or negative. For each snippet of text, the
text classification model outputs a score that indicates the confidence of the
text being correctly classified as either positive or negative.
For more information on how the models in this tutorial are generated, refer to
the
[Text classification with TensorFlow Lite Model Maker](https://ai.google.dev/edge/litert/libraries/modify/text_classification)
tutorial.
## Models and dataset
This tutorial uses models that were trained using the
[SST-2](https://nlp.stanford.edu/sentiment/index.html) (Stanford Sentiment
Treebank) dataset. SST-2 contains 67,349 movie reviews for training and 872
movie reviews for testing, with each review categorized as either positive or
negative. The models used in this app were trained using the TensorFlow Lite
[Model Maker](https://ai.google.dev/edge/litert/libraries/modify/text_classification)
tool.
The example application uses the following pre-trained models:
* [Average Word Vector](https://www.tensorflow.org/lite/inference_with_metadata/task_library/nl_classifier)
(`NLClassifier`) - The Task Library's `NLClassifier` classifies input text
into different categories, and can handle most text classification models.
* [MobileBERT](https://www.tensorflow.org/lite/inference_with_metadata/task_library/bert_nl_classifier)
(`BertNLClassifier`) - The Task Library's `BertNLClassifier` is similar to
the NLClassifier but is tailored for cases that require out-of-graph
Wordpiece and Sentencepiece tokenizations.
## Setup and run the example app
To setup the text classification application, download the example app from
[GitHub](https://github.com/tensorflow/examples/tree/master/lite/examples/text_classification/android)
and run it using [Android Studio](https://developer.android.com/studio/).
### System requirements
* **[Android Studio](https://developer.android.com/studio/index.html)**
version 2021.1.1 (Bumblebee) or higher.
* Android SDK version 31 or higher
* Android device with a minimum OS version of SDK 21 (Android 7.0 - Nougat)
with
[developer mode](https://developer.android.com/studio/debug/dev-options)
enabled, or an Android Emulator.
### Get the example code
Create a local copy of the example code. You will use this code to create a
project in Android Studio and run the example application.
To clone and setup the example code:
1. Clone the git repository
<pre class="devsite-click-to-copy">
git clone https://github.com/tensorflow/examples.git
</pre>
2. Optionally, configure your git instance to use sparse checkout, so you have only
the files for the text classification example app:
<pre class="devsite-click-to-copy">
cd examples
git sparse-checkout init --cone
git sparse-checkout set lite/examples/text_classification/android
</pre>
### Import and run the project
Create a project from the downloaded example code, build the project, and then
run it.
To import and build the example code project:
1. Start [Android Studio](https://developer.android.com/studio).
1. From the Android Studio, select **File > New > Import Project**.
1. Navigate to the example code directory containing the build.gradle file
(`.../examples/lite/examples/text_classification/android/build.gradle`) and
select that directory.
1. If Android Studio requests a Gradle Sync, choose OK.
1. Ensure that your Android device is connected to your computer and developer
mode is enabled. Click the green `Run` arrow.
If you select the correct directory, Android Studio creates a new project and
builds it. This process can take a few minutes, depending on the speed of your
computer and if you have used Android Studio for other projects. When the build
completes, the Android Studio displays a `BUILD SUCCESSFUL` message in the
**Build Output** status panel.
To run the project:
1. From Android Studio, run the project by selecting **Run > Run…**.
1. Select an attached Android device (or emulator) to test the app.
### Using the application
![Text classification example app in Android](../../../images/lite/android/text-classification-screenshot.png){: .attempt-right width="250px"}
After running the project in Android Studio, the application will automatically
open on the connected device or device emulator.
To use the text classifier:
1. Enter a snippet of text in the text box.
1. From the **Delegate** drop-down, choose either `CPU` or `NNAPI`.
1. Specify a model by choosing either `AverageWordVec` or `MobileBERT`.
1. Choose **Classify**.
The application outputs a *positive* score and a *negative* score. These two
scores will sum to 1, and measures the likelihood that the sentiment of the
input text is positive or negative. A higher number denotes a higher level of
confidence.
You now have a functioning text classification application. Use the following
sections to better understand how the example application works, and how to
implement text classification features to your production applications:
* [How the application works](#how_it_works) - A walkthrough of the structure
and key files of the example application.
* [Modify your application](#modify_applications) - Instructions on adding
text classification to an existing application.
## How the example app works {:#how_it_works}
The application uses the
[Task library for natural language (NL)](https://ai.google.dev/edge/litert/libraries/task_library/overview)
package to implement the text classification models. The two models, Average
Word Vector and MobileBERT, were trained using the TensorFlow Lite
[Model Maker](https://ai.google.dev/edge/litert/libraries/modify/text_classification).
The application runs on CPU by default, with the option of hardware acceleration
using the NNAPI delegate.
The following files and directories contain the crucial code for this text
classification application:
* [TextClassificationHelper.kt](https://github.com/tensorflow/examples/blob/master/lite/examples/text_classification/android/app/src/main/java/org/tensorflow/lite/examples/textclassification/TextClassificationHelper.kt) -
Initializes the text classifier and handles the model and delegate
selection.
* [MainActivity.kt](https://github.com/tensorflow/examples/blob/master/lite/examples/text_classification/android/app/src/main/java/org/tensorflow/lite/examples/textclassification/MainActivity.kt) -
Implements the application, including calling `TextClassificationHelper` and
`ResultsAdapter`.
* [ResultsAdapter.kt](https://github.com/tensorflow/examples/blob/master/lite/examples/text_classification/android/app/src/main/java/org/tensorflow/lite/examples/textclassification/ResultsAdapter.kt) -
Handles and formats the results.
## Modify your application {:#modify_applications}
The following sections explain the key steps to modify your own Android app to
run the model shown in the example app. These instructions use the example app
as a reference point. The specific changes needed for your own app may vary from
the example app.
### Open or create an Android project
You need an Android development project in Android Studio to follow along with
the rest of these instructions. Follow the instructions below to open an
existing project or create a new one.
To open an existing Android development project:
* In Android Studio, select *File > Open* and select an existing project.
To create a basic Android development project:
* Follow the instructions in Android Studio to
[Create a basic project](https://developer.android.com/studio/projects/create-project).
For more information on using Android Studio, refer to the
[Android Studio documentation](https://developer.android.com/studio/intro).
### Add project dependencies
In your own application, you must add specific project dependencies to run
TensorFlow Lite machine learning models, and access utility functions that
convert data such as strings, into a tensor data format that can be processed by
the model you are using.
The following instructions explain how to add the required project and module
dependencies to your own Android app project.
To add module dependencies:
1. In the module that uses TensorFlow Lite, update the module's `build.gradle`
file to include the following dependencies.
In the example application, the dependencies are located in
[app/build.gradle](https://github.com/tensorflow/examples/blob/master/lite/examples/text_classification/android/app/build.gradle):
```
dependencies {
...
implementation 'org.tensorflow:tensorflow-lite-task-text:0.4.0'
}
```
The project must include the Text task library
(`tensorflow-lite-task-text`).
If you want to modify this app to run on a graphics processing unit (GPU),
the GPU library (`tensorflow-lite-gpu-delegate-plugin`) provides the
infrastructure to run the app on GPU, and Delegate (`tensorflow-lite-gpu`)
provides the compatibility list. Running this app on GPU is outside the
scope of this tutorial.
1. In Android Studio, sync the project dependencies by selecting: **File > Sync
Project with Gradle Files**.
### Initialize the ML models {:#initialize_models}
In your Android app, you must initialize the TensorFlow Lite machine learning
model with parameters before running predictions with the model.
A TensorFlow Lite model is stored as a `*.tflite` file. The model file contains
the prediction logic and typically includes
[metadata](https://ai.google.dev/edge/litert/models/metadata) about how to
interpret prediction results, such as prediction class names. Typically, model
files are stored in the `src/main/assets` directory of your development project,
as in the code example:
- `<project>/src/main/assets/mobilebert.tflite`
- `<project>/src/main/assets/wordvec.tflite`
Note: The example app uses a
`[download_model.gradle](https://github.com/tensorflow/examples/blob/master/lite/examples/text_classification/android/app/download_model.gradle)`
file to download the
[Average Word Vector](https://www.tensorflow.org/lite/inference_with_metadata/task_library/nl_classifier)
and
[MobileBERT](https://www.tensorflow.org/lite/inference_with_metadata/task_library/bert_nl_classifier)
models at build time. This approach is not necessary or recommended for a
production app.
For convenience and code readability, the example declares a companion object
that defines the settings for the model.
To initialize the model in your app:
1. Create a companion object to define the settings for the model. In the
example application, this object is located in
[TextClassificationHelper.kt](https://github.com/tensorflow/examples/blob/master/lite/examples/text_classification/android/app/src/main/java/org/tensorflow/lite/examples/textclassification/TextClassificationHelper.kt):
```
companion object {
const val DELEGATE_CPU = 0
const val DELEGATE_NNAPI = 1
const val WORD_VEC = "wordvec.tflite"
const val MOBILEBERT = "mobilebert.tflite"
}
```
1. Create the settings for the model by building a classifier object, and
construct a TensorFlow Lite object using either `BertNLClassifier` or
`NLClassifier`.
In the example application, this is located in the `initClassifier` function
within
[TextClassificationHelper.kt](https://github.com/tensorflow/examples/blob/master/lite/examples/text_classification/android/app/src/main/java/org/tensorflow/lite/examples/textclassification/TextClassificationHelper.kt):
```
fun initClassifier() {
...
if( currentModel == MOBILEBERT ) {
...
bertClassifier = BertNLClassifier.createFromFileAndOptions(
context,
MOBILEBERT,
options)
} else if (currentModel == WORD_VEC) {
...
nlClassifier = NLClassifier.createFromFileAndOptions(
context,
WORD_VEC,
options)
}
}
```
Note: Most production apps using text classification will use either
`BertNLClassifier` or `NLClassifier` - not both.
### Enable hardware acceleration (optional) {:#hardware_acceleration}
When initializing a TensorFlow Lite model in your app, you should consider using
hardware acceleration features to speed up the prediction calculations of the
model. TensorFlow Lite
[delegates](https://www.tensorflow.org/lite/performance/delegates) are software
modules that accelerate execution of machine learning models using specialized
processing hardware on a mobile device, such as graphics processing unit (GPUs)
or tensor processing units (TPUs).
To enable hardware acceleration in your app:
1. Create a variable to define the delegate that the application will use. In
the example application, this variable is located early in
[TextClassificationHelper.kt](https://github.com/tensorflow/examples/blob/master/lite/examples/text_classification/android/app/src/main/java/org/tensorflow/lite/examples/textclassification/TextClassificationHelper.kt):
```
var currentDelegate: Int = 0
```
1. Create a delegate selector. In the example application, the delegate
selector is located in the `initClassifier` function within
[TextClassificationHelper.kt](https://github.com/tensorflow/examples/blob/master/lite/examples/text_classification/android/app/src/main/java/org/tensorflow/lite/examples/textclassification/TextClassificationHelper.kt):
```
val baseOptionsBuilder = BaseOptions.builder()
when (currentDelegate) {
DELEGATE_CPU -> {
// Default
}
DELEGATE_NNAPI -> {
baseOptionsBuilder.useNnapi()
}
}
```
Note: It is possible to modify this app to use a GPU delegate, but this requires
that the classifier be created on the same thread that is using the classifier.
This is outside of the scope of this tutorial.
Using delegates for running TensorFlow Lite models is recommended, but not
required. For more information about using delegates with TensorFlow Lite, see
[TensorFlow Lite Delegates](https://www.tensorflow.org/lite/performance/delegates).
### Prepare data for the model
In your Android app, your code provides data to the model for interpretation by
transforming existing data such as raw text into a
[Tensor](https://www.tensorflow.org/lite/api_docs/java/org/tensorflow/lite/Tensor)
data format that can be processed by your model. The data in a Tensor you pass
to a model must have specific dimensions, or shape, that matches the format of
data used to train the model.
This text classification app accepts a
[string](https://developer.android.com/reference/java/lang/String.html) as
input, and the models are trained exclusively on an English language corpus.
Special characters and non-English words are ignored during inference.
To provide text data to the model:
1. Ensure that the `initClassifier` function contains the code for the delegate
and models, as explained in the
[Initialize the ML models](#initialize_models) and
[Enable hardware acceleration](#hardware_acceleration) sections.
1. Use the `init` block to call the `initClassifier` function. In the example
application, the `init` is located in
[TextClassificationHelper.kt](https://github.com/tensorflow/examples/blob/master/lite/examples/text_classification/android/app/src/main/java/org/tensorflow/lite/examples/textclassification/TextClassificationHelper.kt):
```
init {
initClassifier()
}
```
### Run predictions
In your Android app, once you have initialized either a
[BertNLClassifier](https://www.tensorflow.org/lite/api_docs/java/org/tensorflow/lite/task/text/nlclassifier/BertNLClassifier)
or
[NLClassifier](https://www.tensorflow.org/lite/api_docs/java/org/tensorflow/lite/task/text/nlclassifier/NLClassifier)
object, you can begin feeding input text for the model to categorize as
"positive" or "negative".
To run predictions:
1. Create a `classify` function, which uses the selected classifier
(`currentModel`) and measures the time taken to classify the input text
(`inferenceTime`). In the example application, the `classify` function is
located in
[TextClassificationHelper.kt](https://github.com/tensorflow/examples/blob/master/lite/examples/text_classification/android/app/src/main/java/org/tensorflow/lite/examples/textclassification/TextClassificationHelper.kt):
```
fun classify(text: String) {
executor = ScheduledThreadPoolExecutor(1)
executor.execute {
val results: List<Category>
// inferenceTime is the amount of time, in milliseconds, that it takes to
// classify the input text.
var inferenceTime = SystemClock.uptimeMillis()
// Use the appropriate classifier based on the selected model
if(currentModel == MOBILEBERT) {
results = bertClassifier.classify(text)
} else {
results = nlClassifier.classify(text)
}
inferenceTime = SystemClock.uptimeMillis() - inferenceTime
listener.onResult(results, inferenceTime)
}
}
```
1. Pass the results from `classify` to the listener object.
```
fun classify(text: String) {
...
listener.onResult(results, inferenceTime)
}
```
### Handle model output
After you input a line of text, the model produces a prediction score, expressed
as a Float, between 0 and 1 for the 'positive' and 'negative' categories.
To get the prediction results from the model:
1. Create an `onResult` function for the listener object to handle the output.
In the example application, the listener object is located in
[MainActivity.kt](https://github.com/tensorflow/examples/blob/master/lite/examples/text_classification/android/app/src/main/java/org/tensorflow/lite/examples/textclassification/MainActivity.kt)
```
private val listener = object : TextClassificationHelper.TextResultsListener {
override fun onResult(results: List<Category>, inferenceTime: Long) {
runOnUiThread {
activityMainBinding.bottomSheetLayout.inferenceTimeVal.text =
String.format("%d ms", inferenceTime)
adapter.resultsList = results.sortedByDescending {
it.score
}
adapter.notifyDataSetChanged()
}
}
...
}
```
1. Add an `onError` function to the listener object to handle errors:
```
private val listener = object : TextClassificationHelper.TextResultsListener {
...
override fun onError(error: String) {
Toast.makeText(this@MainActivity, error, Toast.LENGTH_SHORT).show()
}
}
```
Once the model has returned a set of prediction results, your application can
act on those predictions by presenting the result to your user or executing
additional logic. The example application lists the prediction scores in the
user interface.
## Next steps
* Train and implement the models from scratch with the
[Text classification with TensorFlow Lite Model Maker](https://ai.google.dev/edge/litert/libraries/modify/text_classification)
tutorial.
* Explore more
[text processing tools for TensorFlow](https://www.tensorflow.org/text).
* Download other BERT models on
[TensorFlow Hub](https://tfhub.dev/google/collections/bert/1).
* Explore various uses of TensorFlow Lite in the [examples](../../examples).
* Learn more about using machine learning models with TensorFlow Lite in the
[Models](../../models) section.
* Learn more about implementing machine learning in your mobile application in
the [TensorFlow Lite Developer Guide](../../guide).