wehub/tensorflow--tensorflow
1
0
Fork 0
Code Issues Pull Requests Actions 5 Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution
cffconvert / validate (push) Has been skipped
Details
License Check / license-check (push) Failing after 2s
Details

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 12:14:16 +08:00
commit 8a852e4b4e
36502 changed files with 9277225 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tensorflow/lite/g3doc/android/tutorials/audio_classification.md
+402
View File
@@ -0,0 +1,402 @@
# Sound and word recognition for Android
This tutorial shows you how to use TensorFlow Lite with pre-built machine
learning models to recognize sounds and spoken words in an Android app. Audio
classification models like the ones shown in this tutorial can be used to detect
activity, identify actions, or recognize voice commands.
![Audio recognition animated demo](https://storage.googleapis.com/download.tensorflow.org/tflite/examples/audio_classification.gif){: .attempt-right}
This tutorial shows you how to download the example code, load the project into
[Android Studio](https://developer.android.com/studio/), and explains key parts
of the code example so you can start adding this functionality to your own app.
The example app code uses the TensorFlow
[Task Library for Audio](https://www.tensorflow.org/lite/inference_with_metadata/task_library/audio_classifier),
which handles most of the audio data recording and preprocessing. For more
information on how audio is pre-processed for use with machine learning models,
see
[Audio Data Preparation and Augmentation](https://www.tensorflow.org/io/tutorials/audio).
## Audio classification with machine learning
The machine learning model in this tutorial recognizes sounds or words from
audio samples recorded with a microphone on an Android device. The example app
in this tutorial allows you to switch between the
[YAMNet/classifier](https://tfhub.dev/google/lite-model/yamnet/classification/tflite/1),
a model that recognizes sounds, and a model that recognizes specific spoken
words, that was
[trained](https://ai.google.dev/edge/litert/libraries/modify/speech_recognition)
using the TensorFlow Lite
[Model Maker](https://www.tensorflow.org/lite/models/modify/model_maker) tool.
The models run predictions on audio clips that contain 15600 individual samples
per clip and are about 1 second in length.
## Setup and run example
For the first part of this tutorial, you download the sample from GitHub and run
it using Android Studio. The following sections of this tutorial explore the
relevant sections of the example, so you can apply them to your own Android
apps.
### System requirements
* [Android Studio](https://developer.android.com/studio/index.html)
version 2021.1.1 (Bumblebee) or higher.
* Android SDK version 31 or higher
* Android device with a minimum OS version of SDK 24 (Android 7.0 -
Nougat) with developer mode enabled.
### Get the example code
Create a local copy of the example code. You will use this code to create a
project in Android Studio and run the sample application.
To clone and setup the example code:
1. Clone the git repository
<pre class="devsite-click-to-copy">
git clone https://github.com/tensorflow/examples.git
</pre>
1. Optionally, configure your git instance to use sparse checkout, so you
have only the files for the example app:
```
cd examples
git sparse-checkout init --cone
git sparse-checkout set lite/examples/audio_classification/android
```
### Import and run the project
Create a project from the downloaded example code, build the project, and then
run it.
To import and build the example code project:
1. Start [Android Studio](https://developer.android.com/studio).
1. In Android Studio, choose **File > New > Import Project**.
1. Navigate to the example code directory containing the `build.gradle` file
(`.../examples/lite/examples/audio_classification/android/build.gradle`)
and select that directory.
If you select the correct directory, Android Studio creates a new project and
builds it. This process can take a few minutes, depending on the speed of your
computer and if you have used Android Studio for other projects. When the build
completes, the Android Studio displays a `BUILD SUCCESSFUL` message in the
**Build Output** status panel.
To run the project:
1. From Android Studio, run the project by selecting **Run > Run 'app'**.
1. Select an attached Android device with a microphone to test the app.
Note: If you use an emulator to run the app, make sure you
[enable audio input](https://developer.android.com/studio/releases/emulator#29.0.6-host-audio)
from the host machine.
The next sections show you the modifications you need to make to your existing
project to add this functionality to your own app, using this example app as a
reference point.
## Add project dependencies
In your own application, you must add specific project dependencies to run
TensorFlow Lite machine learning models, and access utility functions that
convert standard data formats, such as audio, into a tensor data format that can
be processed by the model you are using.
The example app uses the following TensorFlow Lite libraries:
- [TensorFlow Lite Task library Audio API](https://www.tensorflow.org/lite/api_docs/java/org/tensorflow/lite/task/audio/classifier/package-summary) -
Provides the required audio data input classes, execution of the machine
learning model, and output results from the model processing.
The following instructions show how to add the required project dependencies to
your own Android app project.
To add module dependencies:
1. In the module that uses TensorFlow Lite, update the module's
`build.gradle` file to include the following dependencies. In the example
code, this file is located here:
`.../examples/lite/examples/audio_classification/android/build.gradle`
```
dependencies {
...
implementation 'org.tensorflow:tensorflow-lite-task-audio'
}
```
1. In Android Studio, sync the project dependencies by selecting: **File >
Sync Project with Gradle Files**.
## Initialize the ML model
In your Android app, you must initialize the TensorFlow Lite machine learning
model with parameters before running predictions with the model. These
initialization parameters are dependent on the model and can include settings
such as default minimum accuracy thresholds for predictions and labels for words
or sounds that the model can recognize.
A TensorFlow Lite model includes a `*.tflite` file containing the model. The
model file contains the prediction logic and typically includes
[metadata](../../models/convert/metadata.md) about how to interpret prediction
results, such as prediction class names. Model files should be stored in the
`src/main/assets` directory of your development project, as in the code example:
- `<project>/src/main/assets/yamnet.tflite`
For convenience and code readability, the example declares a companion object
that defines the settings for the model.
To initialize the model in your app:
1. Create a companion object to define the settings for the model:
```
companion object {
const val DISPLAY_THRESHOLD = 0.3f
const val DEFAULT_NUM_OF_RESULTS = 2
const val DEFAULT_OVERLAP_VALUE = 0.5f
const val YAMNET_MODEL = "yamnet.tflite"
const val SPEECH_COMMAND_MODEL = "speech.tflite"
}
```
1. Create the settings for the model by building an
`AudioClassifier.AudioClassifierOptions` object:
```
val options = AudioClassifier.AudioClassifierOptions.builder()
.setScoreThreshold(classificationThreshold)
.setMaxResults(numOfResults)
.setBaseOptions(baseOptionsBuilder.build())
.build()
```
1. Use this settings object to construct a TensorFlow Lite
[`AudioClassifier`](https://www.tensorflow.org/lite/inference_with_metadata/task_library/audio_classifier)
object that contains the model:
```
classifier = AudioClassifier.createFromFileAndOptions(context, "yamnet.tflite", options)
```
### Enable hardware acceleration
When initializing a TensorFlow Lite model in your app, you should consider using
hardware acceleration features to speed up the prediction calculations of the
model. TensorFlow Lite
[delegates](https://www.tensorflow.org/lite/performance/delegates) are software
modules that accelerate execution of machine learning models using specialized
processing hardware on a mobile device, such as graphics processing unit (GPUs)
or tensor processing units (TPUs). The code example uses the NNAPI Delegate to
handle hardware acceleration of the model execution:
```
val baseOptionsBuilder = BaseOptions.builder()
.setNumThreads(numThreads)
...
when (currentDelegate) {
DELEGATE_CPU -> {
// Default
}
DELEGATE_NNAPI -> {
baseOptionsBuilder.useNnapi()
}
}
```
Using delegates for running TensorFlow Lite models is recommended, but not
required. For more information about using delegates with TensorFlow Lite, see
[TensorFlow Lite Delegates](https://www.tensorflow.org/lite/performance/delegates).
## Prepare data for the model
In your Android app, your code provides data to the model for interpretation by
transforming existing data such as audio clips into a
[Tensor](https://www.tensorflow.org/lite/api_docs/java/org/tensorflow/lite/Tensor)
data format that can be processed by your model. The data in a Tensor you pass
to a model must have specific dimensions, or shape, that matches the format of
data used to train the model.
The
[YAMNet/classifier model](https://tfhub.dev/google/lite-model/yamnet/classification/tflite/1)
and the customized
[speech commands](https://ai.google.dev/edge/litert/libraries/modify/speech_recognition)
models used in this code example accepts Tensor data objects that represent
single-channel, or mono, audio clips recorded at 16kHz in 0.975 second clips
(15600 samples). Running predictions on new audio data, your app must transform
that audio data into Tensor data objects of that size and shape. The TensorFlow
Lite Task Library
[Audio API](https://www.tensorflow.org/lite/inference_with_metadata/task_library/audio_classifier)
handles the data transformation for you.
In the example code `AudioClassificationHelper` class, the app records live
audio from the device microphones using an Android
[AudioRecord](https://developer.android.com/reference/android/media/AudioRecord)
object. The code uses
[AudioClassifier](https://www.tensorflow.org/lite/api_docs/java/org/tensorflow/lite/task/audio/classifier/AudioClassifier)
to build and configure that object to record audio at a sampling rate
appropriate for the model. The code also uses AudioClassifier to build a
[TensorAudio](https://www.tensorflow.org/lite/api_docs/java/org/tensorflow/lite/support/audio/TensorAudio)
object to store the transformed audio data. Then the TensorAudio object is
passed to the model for analysis.
To provide audio data to the ML model:
- Use the `AudioClassifier` object to create a `TensorAudio` object and a
`AudioRecord` object:
```
fun initClassifier() {
...
try {
classifier = AudioClassifier.createFromFileAndOptions(context, currentModel, options)
// create audio input objects
tensorAudio = classifier.createInputTensorAudio()
recorder = classifier.createAudioRecord()
}
```
Note: Your app must request permission to record audio using an Android device
microphone. See the `fragments/PermissionsFragment` class in the project for an
example. For more information on requesting permissions, see
[Permissions on Android](https://developer.android.com/guide/topics/permissions/overview).
## Run predictions
In your Android app, once you have connected an
[AudioRecord](https://developer.android.com/reference/android/media/AudioRecord)
object and a
[TensorAudio](https://www.tensorflow.org/lite/api_docs/java/org/tensorflow/lite/support/audio/TensorAudio)
object to an AudioClassifier object, you can run the model against that data to
produce a prediction, or *inference*. The example code for this tutorial runs
predictions on clips from a live-recorded audio input stream at a specific
rate.
Model execution consumes significant resources, so it's important to run ML
model predictions on a separate, background thread. The example app uses a
`[ScheduledThreadPoolExecutor](https://developer.android.com/reference/java/util/concurrent/ScheduledThreadPoolExecutor)`
object to isolate the model processing from other functions of the app.
Audio classification models that recognize sounds with a clear beginning and
end, such as words, can produce more accurate predictions on an incoming audio
stream by analyzing overlapping audio clips. This approach
helps the model avoid missing predictions for words that are cut off at the end
of a clip. In the example app, each time you run a prediction the code grabs the
latest 0.975 second clip from the audio recording buffer and analyzes it. You
can make the model analyze overlapping audio clips by setting the model analysis
thread execution pool `interval` value to a length that's shorter than the
length of the clips being analyzed. For example, if your model analyzes 1 second
clips and you set the interval to 500 milliseconds, the model will analyze the
last half of the previous clip and 500 milliseconds of new audio data each time,
creating a clip analysis overlap of 50%.
To start running predictions on the audio data:
1. Use the `AudioClassificationHelper.startAudioClassification()` method
to start the audio recording for the model:
```
fun startAudioClassification() {
if (recorder.recordingState == AudioRecord.RECORDSTATE_RECORDING) {
return
}
recorder.startRecording()
}
```
1. Set how frequently the model generates an inference from the audio
clips by setting a fixed rate `interval` in the
`ScheduledThreadPoolExecutor` object:
```
executor = ScheduledThreadPoolExecutor(1)
executor.scheduleAtFixedRate(
classifyRunnable,
0,
interval,
TimeUnit.MILLISECONDS)
```
1. The `classifyRunnable` object in the code above executes the
`AudioClassificationHelper.classifyAudio()` method, which loads the latest
available audio data from the recorder and performs a prediction:
```
private fun classifyAudio() {
tensorAudio.load(recorder)
val output = classifier.classify(tensorAudio)
...
}
```
Caution: Do not run the ML model predictions on the main execution thread of
your application. Doing so can cause your app user interface to become slow or
unresponsive.
### Stop prediction processing
Make sure your app code stops doing audio classification when your app's audio
processing Fragment or Activity loses focus. Running a machine learning model
continuously has a significant impact on the battery life of an Android device.
Use the `onPause()` method of the Android activity or fragment associated with
the audio classification to stop audio recording and prediction processing.
To stop audio recording and classification:
- Use the `AudioClassificationHelper.stopAudioClassification()` method to
stop recording and the model execution, as shown below in the
`AudioFragment` class:
```
override fun onPause() {
super.onPause()
if (::audioHelper.isInitialized ) {
audioHelper.stopAudioClassification()
}
}
```
## Handle model output
In your Android app, after you process an audio clip, the model produces a list
of predictions which your app code must handle by executing additional business
logic, displaying results to the user, or taking other actions. The output of
any given TensorFlow Lite model varies in terms of the number of predictions it
produces (one or many), and the descriptive information for each prediction. In
the case of the models in the example app, the predictions are either a list of
recognized sounds or words. The AudioClassifier options object used in the code
example lets you set the maximum number of predictions with the
`setMaxResults()` method, as shown in
[Initialize the ML model](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/android/tutorials/audio_classification.md#initialize-the-ml-model)
section.
To get the prediction results from the model:
1. Get the results of the
[AudioClassifier](https://www.tensorflow.org/lite/api_docs/java/org/tensorflow/lite/task/audio/classifier/AudioClassifier)
object's `classify()` method and pass them to the listener object (code
reference):
```
private fun classifyAudio() {
...
val output = classifier.classify(tensorAudio)
listener.onResult(output[0].categories, inferenceTime)
}
```
1. Use the listener's onResult() function handle the output by executing
business logic or displaying results to the user:
```
private val audioClassificationListener = object : AudioClassificationListener {
override fun onResult(results: List<Category>, inferenceTime: Long) {
requireActivity().runOnUiThread {
adapter.categoryList = results
adapter.notifyDataSetChanged()
fragmentAudioBinding.bottomSheetLayout.inferenceTimeVal.text =
String.format("%d ms", inferenceTime)
}
}
```
The model used in this example generates a list of predictions with a label
for the classified sound or word, and a prediction score between 0 and 1 as a
Float representing the confidence of the prediction, with 1 being the highest
confidence rating. In general, predictions with a score below 50% (0.5) are
considered inconclusive. However, how you handle low-value prediction results is
up to you and the needs of your application.
Once the model has returned a set of prediction results, your application can
act on those predictions by presenting the result to your user or executing
additional logic. In the case of the example code, the application lists the
identified sounds or words in the app user interface.
## Next steps
You can find additional TensorFlow Lite models for audio processing on
[TensorFlow Hub](https://tfhub.dev/s?deployment-format=lite&module-type=audio-embedding,audio-pitch-extraction,audio-event-classification,audio-stt)
and through the
[Pre-trained models guide](https://www.tensorflow.org/lite/models/trained) page.
For more information about implementing machine learning in your mobile
application with TensorFlow Lite, see the
[TensorFlow Lite Developer Guide](https://www.tensorflow.org/lite/guide).
tensorflow/lite/g3doc/android/tutorials/object_detection.md
+491
View File
@@ -0,0 +1,491 @@
# Object detection with Android
This tutorial shows you how to build an Android app using TensorFlow Lite to
continuously detect objects in frames captured by a device camera. This
application is designed for a physical Android device. If you are updating an
existing project, you can use the code sample as a reference and skip ahead to
the instructions for [modifying your project](#add_dependencies).
![Object detection animated demo](https://storage.googleapis.com/download.tensorflow.org/tflite/examples/obj_detection_cat.gif){: .attempt-right width="250px"}
## Object detection overview
*Object detection* is the machine learning task of identifying the presence and
location of multiple classes of objects within an image. An object detection
model is trained on a dataset that contains a set of known objects.
The trained model receives image frames as input and attempts to categorize
items in the images from the set of known classes it was trained to recognize.
For each image frame, the object detection model outputs a list of the objects
it detects, the location of a bounding box for each object, and a score that
indicates the confidence of the object being correctly classified.
## Models and dataset
This tutorial uses models that were trained using the
[COCO dataset](http://cocodataset.org/). COCO is a large-scale object detection
dataset that contains 330K images, 1.5 million object instances, and 80 object
categories.
You have the option to use one of the following pre-trained models:
* [EfficientDet-Lite0](https://tfhub.dev/tensorflow/lite-model/efficientdet/lite0/detection/metadata/1)
*[Recommended]* - a lightweight object detection model with a BiFPN feature
extractor, shared box predictor, and focal loss. The mAP (mean Average
Precision) for the COCO 2017 validation dataset is 25.69%.
* [EfficientDet-Lite1](https://tfhub.dev/tensorflow/lite-model/efficientdet/lite1/detection/metadata/1) -
a medium-sized EfficientDet object detection model. The mAP for the COCO
2017 validation dataset is 30.55%.
* [EfficientDet-Lite2](https://tfhub.dev/tensorflow/lite-model/efficientdet/lite2/detection/metadata/1) -
a larger EfficientDet object detection model. The mAP for the COCO 2017
validation dataset is 33.97%.
* [MobileNetV1-SSD](https://tfhub.dev/tensorflow/lite-model/ssd_mobilenet_v1/1/metadata/2) -
an extremely lightweight model optimized to work with TensorFlow Lite for
object detection. The mAP for the COCO 2017 validation dataset is 21%.
For this tutorial, the *EfficientDet-Lite0* model strikes a good balance between
size and accuracy.
Downloading, extraction, and placing the models into the assets folder is
managed automatically by the `download.gradle` file, which is run at build time.
You don't need to manually download TFLite models into the project.
## Setup and run example
To setup the object detection app, download the sample from
[GitHub](https://github.com/tensorflow/examples/tree/master/lite/examples/object_detection/android)
and run it using [Android Studio](https://developer.android.com/studio/). The
following sections of this tutorial explore the relevant sections of the code
example, so you can apply them to your own Android apps.
### System requirements
* **[Android Studio](https://developer.android.com/studio/index.html)**
version 2021.1.1 (Bumblebee) or higher.
* Android SDK version 31 or higher
* Android device with a minimum OS version of SDK 24 (Android 7.0 -
Nougat) with developer mode enabled.
Note: This example uses a camera, so run it on a physical Android device.
### Get the example code
Create a local copy of the example code. You will use this code to create a
project in Android Studio and run the sample application.
To clone and setup the example code:
1. Clone the git repository
<pre class="devsite-click-to-copy">
git clone https://github.com/tensorflow/examples.git
</pre>
2. Optionally, configure your git instance to use sparse checkout, so you have only
the files for the object detection example app:
<pre class="devsite-click-to-copy">
cd examples
git sparse-checkout init --cone
git sparse-checkout set lite/examples/object_detection/android
</pre>
### Import and run the project
Create a project from the downloaded example code, build the project, and then
run it.
To import and build the example code project:
1. Start [Android Studio](https://developer.android.com/studio).
1. From the Android Studio, select **File > New > Import Project**.
1. Navigate to the example code directory containing the build.gradle file
(`.../examples/lite/examples/object_detection/android/build.gradle`) and
select that directory.
1. If Android Studio requests a Gradle Sync, choose OK.
1. Ensure that your Android device is connected to your computer and developer
mode is enabled. Click the green `Run` arrow.
If you select the correct directory, Android Studio creates a new project and
builds it. This process can take a few minutes, depending on the speed of your
computer and if you have used Android Studio for other projects. When the build
completes, the Android Studio displays a `BUILD SUCCESSFUL` message in the
**Build Output** status panel.
Note: The example code is built with Android Studio 4.2.2, but works with
earlier versions of Studio. If you are using an earlier version of Android
Studio you can try adjust the version number of the Android plugin so that the
build completes, instead of upgrading Studio.
**Optional:** To fix build errors by updating the Android plugin version:
1. Open the build.gradle file in the project directory.
1. Change the Android tools version as follows:
```
// from: classpath
'com.android.tools.build:gradle:4.2.2'
// to: classpath
'com.android.tools.build:gradle:4.1.2'
```
1. Sync the project by selecting: **File > Sync Project with Gradle Files**.
To run the project:
1. From Android Studio, run the project by selecting **Run > Run…**.
1. Select an attached Android device with a camera to test the app.
The next sections show you the modifications you need to make to your existing
project to add this functionality to your own app, using this example app as a
reference point.
## Add project dependencies {:#add_dependencies}
In your own application, you must add specific project dependencies to run
TensorFlow Lite machine learning models, and access utility functions that
convert data such as images, into a tensor data format that can be processed by
the model you are using.
The example app uses the TensorFlow Lite
[Task library for vision](../../inference_with_metadata/task_library/overview.md#supported-tasks)
to enable execution of the object detection machine learning model. The
following instructions explain how to add the required library dependencies to
your own Android app project.
The following instructions explain how to add the required project and module
dependencies to your own Android app project.
To add module dependencies:
1. In the module that uses TensorFlow Lite, update the module's `build.gradle`
file to include the following dependencies. In the example code, this file
is located here:
`...examples/lite/examples/object_detection/android/app/build.gradle`
([code reference](https://github.com/tensorflow/examples/blob/master/lite/examples/object_detection/android/app/build.gradle))
```
dependencies {
...
implementation 'org.tensorflow:tensorflow-lite-task-vision:0.4.0'
// Import the GPU delegate plugin Library for GPU inference
implementation 'org.tensorflow:tensorflow-lite-gpu-delegate-plugin:0.4.0'
implementation 'org.tensorflow:tensorflow-lite-gpu:2.9.0'
}
```
The project must include the Vision task library
(`tensorflow-lite-task-vision`). The graphics processing unit (GPU) library
(`tensorflow-lite-gpu-delegate-plugin`) provides the infrastructure to run
the app on GPU, and Delegate (`tensorflow-lite-gpu`) provides the
compatibility list.
1. In Android Studio, sync the project dependencies by selecting: **File > Sync
Project with Gradle Files**.
## Initialize the ML model
In your Android app, you must initialize the TensorFlow Lite machine learning
model with parameters before running predictions with the model. These
initialization parameters are consistent across object detection models and can
include settings such as minimum accuracy thresholds for predictions.
A TensorFlow Lite model includes a `.tflite` file containing the model code and
frequently includes a labels file containing the names of the classes predicted
by the model. In the case of object detection, classes are objects such as a
person, dog, cat, or car.
This example downloads several models that are specified in
`download_models.gradle`, and the `ObjectDetectorHelper` class provides a
selector for the models:
```
val modelName =
when (currentModel) {
MODEL_MOBILENETV1 -> "mobilenetv1.tflite"
MODEL_EFFICIENTDETV0 -> "efficientdet-lite0.tflite"
MODEL_EFFICIENTDETV1 -> "efficientdet-lite1.tflite"
MODEL_EFFICIENTDETV2 -> "efficientdet-lite2.tflite"
else -> "mobilenetv1.tflite"
}
```
Key Point: Models should be stored in the `src/main/assets` directory of your
development project. The TensorFlow Lite Task library automatically checks this
directory when you specify a model file name.
To initialize the model in your app:
1. Add a `.tflite` model file to the `src/main/assets` directory of your
development project, such as:
[EfficientDet-Lite0](https://tfhub.dev/tensorflow/lite-model/efficientdet/lite0/detection/metadata/1).
1. Set a static variable for your model's file name. In the example app, you
set the `modelName` variable to `MODEL_EFFICIENTDETV0` to use the
EfficientDet-Lite0 detection model.
1. Set the options for model, such as the prediction threshold, results set
size, and optionally, hardware acceleration delegates:
```
val optionsBuilder =
ObjectDetector.ObjectDetectorOptions.builder()
.setScoreThreshold(threshold)
.setMaxResults(maxResults)
```
1. Use the settings from this object to construct a TensorFlow Lite
[`ObjectDetector`](https://www.tensorflow.org/lite/api_docs/java/org/tensorflow/lite/task/vision/detector/ObjectDetector#createFromFile\(Context,%20java.lang.String\))
object that contains the model:
```
objectDetector =
ObjectDetector.createFromFileAndOptions(
context, modelName, optionsBuilder.build())
```
The `setupObjectDetector` sets up the following model parameters:
* Detection threshold
* Maximum number of detection results
* Number of processing threads to use
(`BaseOptions.builder().setNumThreads(numThreads)`)
* Actual model (`modelName`)
* ObjectDetector object (`objectDetector`)
### Configure hardware accelerator
When initializing a TensorFlow Lite model in your application, you can use
hardware acceleration features to speed up the prediction calculations of the
model.
TensorFlow Lite *delegates* are software modules that accelerate execution of
machine learning models using specialized processing hardware on a mobile
device, such as Graphics Processing Units (GPUs), Tensor Processing Units
(TPUs), and Digital Signal Processors (DSPs). Using delegates for running
TensorFlow Lite models is recommended, but not required.
The object detector is initialized using the current settings on the thread that
is using it. You can use CPU and [NNAPI](../../android/delegates/nnapi.md)
delegates with detectors that are created on the main thread and used on a
background thread, but the thread that initialized the detector must use the GPU
delegate.
The delegates are set within the `ObjectDetectionHelper.setupObjectDetector()`
function:
```
when (currentDelegate) {
DELEGATE_CPU -> {
// Default
}
DELEGATE_GPU -> {
if (CompatibilityList().isDelegateSupportedOnThisDevice) {
baseOptionsBuilder.useGpu()
} else {
objectDetectorListener?.onError("GPU is not supported on this device")
}
}
DELEGATE_NNAPI -> {
baseOptionsBuilder.useNnapi()
}
}
```
For more information about using hardware acceleration delegates with TensorFlow
Lite, see [TensorFlow Lite Delegates](../../performance/delegates.md).
## Prepare data for the model
In your Android app, your code provides data to the model for interpretation by
transforming existing data such as image frames into a Tensor data format that
can be processed by your model. The data in a Tensor you pass to a model must
have specific dimensions, or shape, that matches the format of data used to
train the model.
The
[EfficientDet-Lite0](https://tfhub.dev/tensorflow/lite-model/efficientdet/lite0/detection/metadata/1)
model used in this code example accepts Tensors representing images with a
dimension of 320 x 320, with three channels (red, blue, and green) per pixel.
Each value in the tensor is a single byte between 0 and 255. So, to run
predictions on new images, your app must transform that image data into Tensor
data objects of that size and shape. The TensorFlow Lite Task Library Vision API
handles the data transformation for you.
The app uses an
[`ImageAnalysis`](https://developer.android.com/training/camerax/analyze) object
to pull images from the camera. This object calls the `detectObject` function
with bitmap from the camera. The data is automatically resized and rotated by
`ImageProcessor` so that it meets the image data requirements of the model. The
image is then translated into a
[`TensorImage`](https://www.tensorflow.org/lite/api_docs/java/org/tensorflow/lite/support/image/TensorImage)
object.
To prepare data from the camera subsystem to be processed by the ML model:
1. Build an `ImageAnalysis` object to extract images in the required format:
```
imageAnalyzer =
ImageAnalysis.Builder()
.setTargetAspectRatio(AspectRatio.RATIO_4_3)
.setTargetRotation(fragmentCameraBinding.viewFinder.display.rotation)
.setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
.setOutputImageFormat(OUTPUT_IMAGE_FORMAT_RGBA_8888)
.build()
...
```
1. Connect the analyzer to the camera subsystem and create a bitmap buffer to
contain the data received from the camera:
```
.also {
it.setAnalyzer(cameraExecutor) {
image -> if (!::bitmapBuffer.isInitialized)
{ bitmapBuffer = Bitmap.createBitmap( image.width, image.height,
Bitmap.Config.ARGB_8888 ) } detectObjects(image)
}
}
```
1. Extract the specific image data needed by the model, and pass the image
rotation information:
```
private fun detectObjects(image: ImageProxy) {
//Copy out RGB bits to the shared bitmap buffer
image.use {bitmapBuffer.copyPixelsFromBuffer(image.planes[0].buffer) }
val imageRotation = image.imageInfo.rotationDegrees
objectDetectorHelper.detect(bitmapBuffer, imageRotation)
}
```
1. Complete any final data transformations and add the image data to a
`TensorImage` object, as shown in the `ObjectDetectorHelper.detect()` method
of the example app:
```
val imageProcessor = ImageProcessor.Builder().add(Rot90Op(-imageRotation / 90)).build()
// Preprocess the image and convert it into a TensorImage for detection.
val tensorImage = imageProcessor.process(TensorImage.fromBitmap(image))
```
Note: When extracting image information from the Android camera subsystem, make
sure to get an image in RGB format. This format is required by the TensorFlow
Lite ImageProcessor class which you use to prepare the image for analysis by a
model. If the RGB-format image contains an Alpha channel, that transparency data
is ignored.
## Run predictions
In your Android app, once you create a TensorImage object with image data in the
correct format, you can run the model against that data to produce a prediction,
or *inference*.
In the `fragments/CameraFragment.kt` class of the example app, the
`imageAnalyzer` object within the `bindCameraUseCases` function automatically
passes data to the model for predictions when the app is connected to the
camera.
The app uses the `cameraProvider.bindToLifecycle()` method to handle the camera
selector, preview window, and ML model processing. The `ObjectDetectorHelper.kt`
class handles passing the image data into the model. To run the model and
generate predictions from image data:
- Run the prediction by passing the image data to your predict function:
```
val results = objectDetector?.detect(tensorImage)
```
The TensorFlow Lite Interpreter object receives this data, runs it against the
model, and produces a list of predictions. For continuous processing of data by
the model, use the `runForMultipleInputsOutputs()` method so that Interpreter
objects are not created and then removed by the system for each prediction run.
## Handle model output
In your Android app, after you run image data against the object detection
model, it produces a list of predictions which your app code must handle by
executing additional business logic, displaying results to the user, or taking
other actions.
The output of any given TensorFlow Lite model varies in terms of the number of
predictions it produces (one or many), and the descriptive information for each
prediction. In the case of an object detection model, predictions typically
include data for a bounding box that indicates where an object is detected in
the image. In the example code, the results are passed to the `onResults`
function in `CameraFragment.kt`, which is acting as a DetectorListener on the
object detection process.
```
interface DetectorListener {
fun onError(error: String)
fun onResults(
results: MutableList<Detection>?,
inferenceTime: Long,
imageHeight: Int,
imageWidth: Int
)
}
```
For the model used in this example, each prediction includes a bounding box
location for the object, a label for the object, and a prediction score between
0 and 1 as a Float representing the confidence of the prediction, with 1 being
the highest confidence rating. In general, predictions with a score below 50%
(0.5) are considered inconclusive. However, how you handle low-value prediction
results is up to you and the needs of your application.
To handle model prediction results:
1. Use a listener pattern to pass results to your app code or user interface
objects. The example app uses this pattern to pass detection results from
the `ObjectDetectorHelper` object to the `CameraFragment` object:
```
objectDetectorListener.onResults(
// instance of CameraFragment
results,
inferenceTime,
tensorImage.height,
tensorImage.width)
```
1. Act on the results, such as displaying the prediction to the user. The
example draws an overlay on the CameraPreview object to show the result:
```
override fun onResults(
results: MutableList<Detection>?,
inferenceTime: Long,
imageHeight: Int,
imageWidth: Int
) {
activity?.runOnUiThread {
fragmentCameraBinding.bottomSheetLayout.inferenceTimeVal.text =
String.format("%d ms", inferenceTime)
// Pass necessary information to OverlayView for drawing on the canvas
fragmentCameraBinding.overlay.setResults(
results ?: LinkedList<Detection>(),
imageHeight,
imageWidth
)
// Force a redraw
fragmentCameraBinding.overlay.invalidate()
}
}
```
Once the model has returned a prediction result, your application can act on
that prediction by presenting the result to your user or executing additional
logic. In the case of the example code, the application draws a bounding box
around the identified object and displays the class name on screen.
## Next steps
* Explore various uses of TensorFlow Lite in the [examples](../../examples).
* Learn more about using machine learning models with TensorFlow Lite in the
[Models](../../models) section.
* Learn more about implementing machine learning in your mobile application in
the [TensorFlow Lite Developer Guide](../../guide).
tensorflow/lite/g3doc/android/tutorials/question_answer.md
+534
View File
@@ -0,0 +1,534 @@
# Answering questions with Android
![Question answering example app in Android](../../examples/bert_qa/images/screenshot.gif){: .attempt-right width="250px"}
This tutorial shows you how to build an Android application using TensorFlow
Lite to provide answers to questions structured in natural language text. The
[example application](https://github.com/tensorflow/examples/tree/master/lite/examples/bert_qa/android)
uses the *BERT question answerer*
([`BertQuestionAnswerer`](../../inference_with_metadata/task_library/bert_question_answerer))
API within the
[Task library for natural language (NL)](https://ai.google.dev/edge/litert/libraries/task_library/overview)
to enable Question answering machine learning models. The application is
designed for a physical Android device but can also run on a device emulator.
If you are updating an existing project, you can use the example application as
a reference or template. For instructions on how to add question answering to an
existing application, refer to
[Updating and modifying your application](#modify_applications).
## Question answering overview
*Question answering* is the machine learning task of answering questions posed
in natural language. A trained question answering model receives a text passage
and question as input, and attempts to answer the question based on its
interpretation of the information within the passage.
A Question answering model is trained on a question answering dataset, which
consists of a reading comprehension dataset along with question-answer pairs
based on different segments of text.
For more information on how the models in this tutorial are generated, refer to
the
[BERT Question Answer with TensorFlow Lite Model Maker](https://ai.google.dev/edge/litert/libraries/modify/question_answer)
tutorial.
## Models and dataset
The example app uses the Mobile BERT Q&A
([`mobilebert`](https://tfhub.dev/tensorflow/lite-model/mobilebert/1/metadata/1))
model, which is a lighter and faster version of
[BERT](https://arxiv.org/abs/1810.04805) (Bidirectional Encoder Representations
from Transformers). For more information on `mobilebert`, see the
[MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices](https://arxiv.org/abs/2004.02984)
research paper.
The `mobilebert` model was trained using the Stanford Question Answering Dataset
([SQuAD](https://rajpurkar.github.io/SQuAD-explorer/)) dataset, a reading
comprehension dataset consisting of articles from Wikipedia and a set of
question-answer pairs for each article.
## Setup and run the example app
To setup the question answering application, download the example app from
[GitHub](https://github.com/tensorflow/examples/tree/master/lite/examples/bert_qa/android)
and run it using [Android Studio](https://developer.android.com/studio/).
### System requirements
* **[Android Studio](https://developer.android.com/studio/index.html)**
version 2021.1.1 (Bumblebee) or higher.
* Android SDK version 31 or higher
* Android device with a minimum OS version of SDK 21 (Android 7.0 - Nougat)
with
[developer mode](https://developer.android.com/studio/debug/dev-options)
enabled, or an Android Emulator.
### Get the example code
Create a local copy of the example code. You will use this code to create a
project in Android Studio and run the example application.
To clone and setup the example code:
1. Clone the git repository
<pre class="devsite-click-to-copy">
git clone https://github.com/tensorflow/examples.git
</pre>
2. Optionally, configure your git instance to use sparse checkout, so you have only
the files for the question answering example app:
<pre class="devsite-click-to-copy">
cd examples
git sparse-checkout init --cone
git sparse-checkout set lite/examples/bert_qa/android
</pre>
### Import and run the project
Create a project from the downloaded example code, build the project, and then
run it.
To import and build the example code project:
1. Start [Android Studio](https://developer.android.com/studio).
1. From the Android Studio, select **File > New > Import Project**.
1. Navigate to the example code directory containing the build.gradle file
(`.../examples/lite/examples/bert_qa/android/build.gradle`) and select that
directory.
1. If Android Studio requests a Gradle Sync, choose OK.
1. Ensure that your Android device is connected to your computer and developer
mode is enabled. Click the green `Run` arrow.
If you select the correct directory, Android Studio creates a new project and
builds it. This process can take a few minutes, depending on the speed of your
computer and if you have used Android Studio for other projects. When the build
completes, the Android Studio displays a `BUILD SUCCESSFUL` message in the
**Build Output** status panel.
To run the project:
1. From Android Studio, run the project by selecting **Run > Run…**.
1. Select an attached Android device (or emulator) to test the app.
### Using the application
After running the project in Android Studio, the application automatically opens
on the connected device or device emulator.
To use the Question answerer example app:
1. Choose a topic from the list of subjects.
1. Choose a suggested question or enter your own in the text box.
1. Toggle the orange arrow to run the model.
The application attempts to identify the answer to the question from the passage
text. If the model detects an answer within the passage, the application
highlights the relevant span of text for the user.
You now have a functioning question answering application. Use the following
sections to better understand how the example application works, and how to
implement question answering features in your production applications:
* [How the application works](#how_it_works) - A walkthrough of the structure
and key files of the example application.
* [Modify your application](#modify_applications) - Instructions on adding
question answering to an existing application.
## How the example app works {:#how_it_works}
The application uses the `BertQuestionAnswerer` API within the
[Task library for natural language (NL)](https://ai.google.dev/edge/litert/libraries/task_library/overview)
package. The MobileBERT model was trained using the TensorFlow Lite
[Model Maker](https://ai.google.dev/edge/litert/libraries/modify/question_answer).
The application runs on CPU by default, with the option of hardware acceleration
using the GPU or NNAPI delegate.
The following files and directories contain the crucial code for this
application:
* [BertQaHelper.kt](https://github.com/tensorflow/examples/blob/master/lite/examples/bert_qa/android/app/src/main/java/org/tensorflow/lite/examples/bertqa/BertQaHelper.kt) -
Initializes the question answerer and handles the model and delegate
selection.
* [QaFragment.kt](https://github.com/tensorflow/examples/blob/master/lite/examples/bert_qa/android/app/src/main/java/org/tensorflow/lite/examples/bertqa/fragments/QaFragment.kt) -
Handles and formats the results.
* [MainActivity.kt](https://github.com/tensorflow/examples/blob/master/lite/examples/bert_qa/android/app/src/main/java/org/tensorflow/lite/examples/bertqa/MainActivity.kt) -
Provides the organizing logic of the app.
## Modify your application {:#modify_applications}
The following sections explain the key steps to modify your own Android app to
run the model shown in the example app. These instructions use the example app
as a reference point. The specific changes needed for your own app may vary from
the example app.
### Open or create an Android project
You need an Android development project in Android Studio to follow along with
the rest of these instructions. Follow the instructions below to open an
existing project or create a new one.
To open an existing Android development project:
* In Android Studio, select *File > Open* and select an existing project.
To create a basic Android development project:
* Follow the instructions in Android Studio to
[Create a basic project](https://developer.android.com/studio/projects/create-project).
For more information on using Android Studio, refer to the
[Android Studio documentation](https://developer.android.com/studio/intro).
### Add project dependencies
In your own application, add specific project dependencies to run TensorFlow
Lite machine learning models and access utility functions. These functions
convert data such as strings into a tensor data format that can be processed by
the model. The following instructions explain how to add the required project
and module dependencies to your own Android app project.
To add module dependencies:
1. In the module that uses TensorFlow Lite, update the module's `build.gradle`
file to include the following dependencies.
In the example application, the dependencies are located in
[app/build.gradle](https://github.com/tensorflow/examples/blob/master/lite/examples/bert_qa/android/app/build.gradle):
```
dependencies {
...
// Import tensorflow library
implementation 'org.tensorflow:tensorflow-lite-task-text:0.3.0'
// Import the GPU delegate plugin Library for GPU inference
implementation 'org.tensorflow:tensorflow-lite-gpu-delegate-plugin:0.4.0'
implementation 'org.tensorflow:tensorflow-lite-gpu:2.9.0'
}
```
The project must include the Text task library
(`tensorflow-lite-task-text`).
If you want to modify this app to run on a graphics processing unit (GPU),
the GPU library (`tensorflow-lite-gpu-delegate-plugin`) provides the
infrastructure to run the app on GPU, and Delegate (`tensorflow-lite-gpu`)
provides the compatibility list.
1. In Android Studio, sync the project dependencies by selecting: **File > Sync
Project with Gradle Files**.
### Initialize the ML models {:#initialize_models}
In your Android app, you must initialize the TensorFlow Lite machine learning
model with parameters before running predictions with the model.
A TensorFlow Lite model is stored as a `*.tflite` file. The model file contains
the prediction logic and typically includes
[metadata](https://ai.google.dev/edge/litert/models/metadata) about how to
interpret prediction results. Typically, model files are stored in the
`src/main/assets` directory of your development project, as in the code example:
- `<project>/src/main/assets/mobilebert_qa.tflite`
Note: The example app uses a
[`download_model.gradle`](https://github.com/tensorflow/examples/blob/master/lite/examples/bert_qa/android/app/download_models.gradle)
file to download the
[mobilebert_qa](https://www.tensorflow.org/lite/inference_with_metadata/task_library/bert_question_answerer)
model and
[passage text](https://storage.googleapis.com/download.tensorflow.org/models/tflite/bert_qa/contents_from_squad.json)
at build time. This approach is not required for a production app.
For convenience and code readability, the example declares a companion object
that defines the settings for the model.
To initialize the model in your app:
1. Create a companion object to define the settings for the model. In the
example application, this object is located in
[BertQaHelper.kt](https://github.com/tensorflow/examples/blob/master/lite/examples/bert_qa/android/app/src/main/java/org/tensorflow/lite/examples/bertqa/BertQaHelper.kt#L100-L106):
```
companion object {
private const val BERT_QA_MODEL = "mobilebert.tflite"
private const val TAG = "BertQaHelper"
const val DELEGATE_CPU = 0
const val DELEGATE_GPU = 1
const val DELEGATE_NNAPI = 2
}
```
1. Create the settings for the model by building a `BertQaHelper` object, and
construct a TensorFlow Lite object with `bertQuestionAnswerer`.
In the example application, this is located in the
`setupBertQuestionAnswerer()` function within
[BertQaHelper.kt](https://github.com/tensorflow/examples/blob/master/lite/examples/bert_qa/android/app/src/main/java/org/tensorflow/lite/examples/bertqa/BertQaHelper.kt#L41-L76):
```
class BertQaHelper(
...
) {
...
init {
setupBertQuestionAnswerer()
}
fun clearBertQuestionAnswerer() {
bertQuestionAnswerer = null
}
private fun setupBertQuestionAnswerer() {
val baseOptionsBuilder = BaseOptions.builder().setNumThreads(numThreads)
...
val options = BertQuestionAnswererOptions.builder()
.setBaseOptions(baseOptionsBuilder.build())
.build()
try {
bertQuestionAnswerer =
BertQuestionAnswerer.createFromFileAndOptions(context, BERT_QA_MODEL, options)
} catch (e: IllegalStateException) {
answererListener
?.onError("Bert Question Answerer failed to initialize. See error logs for details")
Log.e(TAG, "TFLite failed to load model with error: " + e.message)
}
}
...
}
```
### Enable hardware acceleration (optional) {:#hardware_acceleration}
When initializing a TensorFlow Lite model in your app, you should consider using
hardware acceleration features to speed up the prediction calculations of the
model. TensorFlow Lite
[delegates](https://www.tensorflow.org/lite/performance/delegates) are software
modules that accelerate the execution of machine learning models using
specialized processing hardware on a mobile device, such as graphics processing
unit (GPUs) or tensor processing units (TPUs).
To enable hardware acceleration in your app:
1. Create a variable to define the delegate that the application will use. In
the example application, this variable is located early in
[BertQaHelper.kt](https://github.com/tensorflow/examples/blob/master/lite/examples/bert_qa/android/app/src/main/java/org/tensorflow/lite/examples/bertqa/BertQaHelper.kt#L31):
```
var currentDelegate: Int = 0
```
1. Create a delegate selector. In the example application, the delegate
selector is located in the `setupBertQuestionAnswerer` function within
[BertQaHelper.kt](https://github.com/tensorflow/examples/blob/master/lite/examples/bert_qa/android/app/src/main/java/org/tensorflow/lite/examples/bertqa/BertQaHelper.kt#L48-L62):
```
when (currentDelegate) {
DELEGATE_CPU -> {
// Default
}
DELEGATE_GPU -> {
if (CompatibilityList().isDelegateSupportedOnThisDevice) {
baseOptionsBuilder.useGpu()
} else {
answererListener?.onError("GPU is not supported on this device")
}
}
DELEGATE_NNAPI -> {
baseOptionsBuilder.useNnapi()
}
}
```
Using delegates for running TensorFlow Lite models is recommended, but not
required. For more information about using delegates with TensorFlow Lite, see
[TensorFlow Lite Delegates](https://www.tensorflow.org/lite/performance/delegates).
### Prepare data for the model
In your Android app, your code provides data to the model for interpretation by
transforming existing data such as raw text into a
[Tensor](https://www.tensorflow.org/lite/api_docs/java/org/tensorflow/lite/Tensor)
data format that can be processed by your model. The Tensor you pass to a model
must have specific dimensions, or shape, that matches the format of data used to
train the model. This question answering app accepts
[strings](https://developer.android.com/reference/java/lang/String.html) as
inputs for both the text passage and question. The model does not recognize
special characters and non-English words.
To provide passage text data to the model:
1. Use the `LoadDataSetClient` object to load the passage text data to the app.
In the example application, this is located in
[LoadDataSetClient.kt](https://github.com/tensorflow/examples/blob/master/lite/examples/bert_qa/android/app/src/main/java/org/tensorflow/lite/examples/bertqa/dataset/LoadDataSetClient.kt#L25-L45)
```
fun loadJson(): DataSet? {
var dataSet: DataSet? = null
try {
val inputStream: InputStream = context.assets.open(JSON_DIR)
val bufferReader = inputStream.bufferedReader()
val stringJson: String = bufferReader.use { it.readText() }
val datasetType = object : TypeToken<DataSet>() {}.type
dataSet = Gson().fromJson(stringJson, datasetType)
} catch (e: IOException) {
Log.e(TAG, e.message.toString())
}
return dataSet
}
```
1. Use the `DatasetFragment` object to list the titles for each passage of text
and start the **TFL Question and Answer** screen. In the example
application, this is located in
[DatasetFragment.kt](https://github.com/tensorflow/examples/blob/master/lite/examples/bert_qa/android/app/src/main/java/org/tensorflow/lite/examples/bertqa/fragments/DatasetFragment.kt):
```
class DatasetFragment : Fragment() {
...
override fun onViewCreated(view: View, savedInstanceState: Bundle?) {
super.onViewCreated(view, savedInstanceState)
val client = LoadDataSetClient(requireActivity())
client.loadJson()?.let {
titles = it.getTitles()
}
...
}
...
}
```
1. Use the `onCreateViewHolder` function within the `DatasetAdapter` object to
present the titles for each passage of text. In the example application,
this is located in
[DatasetAdapter.kt](https://github.com/tensorflow/examples/blob/master/lite/examples/bert_qa/android/app/src/main/java/org/tensorflow/lite/examples/bertqa/fragments/DatasetAdapter.kt):
```
override fun onCreateViewHolder(parent: ViewGroup, viewType: Int): ViewHolder {
val binding = ItemDatasetBinding.inflate(
LayoutInflater.from(parent.context),
parent,
false
)
return ViewHolder(binding)
}
```
To provide user questions to the model:
1. Use the `QaAdapter` object to provide the question to the model. In the
example application, this is located in
[QaAdapter.kt](https://github.com/tensorflow/examples/blob/master/lite/examples/bert_qa/android/app/src/main/java/org/tensorflow/lite/examples/bertqa/fragments/QaAdapter.kt):
```
class QaAdapter(private val question: List<String>, private val select: (Int) -> Unit) :
RecyclerView.Adapter<QaAdapter.ViewHolder>() {
inner class ViewHolder(private val binding: ItemQuestionBinding) :
RecyclerView.ViewHolder(binding.root) {
init {
binding.tvQuestionSuggestion.setOnClickListener {
select.invoke(adapterPosition)
}
}
fun bind(question: String) {
binding.tvQuestionSuggestion.text = question
}
}
...
}
```
### Run predictions
In your Android app, once you have initialized a
[BertQuestionAnswerer](https://www.tensorflow.org/lite/api_docs/java/org/tensorflow/lite/task/text/qa/BertQuestionAnswerer)
object, you can begin inputting questions in the form of natural language text
to the model. The model attempts to identify the answer within the text passage.
To run predictions:
1. Create an `answer` function, which runs the model and measures the time
taken to identify the answer (`inferenceTime`). In the example application,
the `answer` function is located in
[BertQaHelper.kt](https://github.com/tensorflow/examples/blob/master/lite/examples/bert_qa/android/app/src/main/java/org/tensorflow/lite/examples/bertqa/BertQaHelper.kt#L78-L98):
```
fun answer(contextOfQuestion: String, question: String) {
if (bertQuestionAnswerer == null) {
setupBertQuestionAnswerer()
}
var inferenceTime = SystemClock.uptimeMillis()
val answers = bertQuestionAnswerer?.answer(contextOfQuestion, question)
inferenceTime = SystemClock.uptimeMillis() - inferenceTime
answererListener?.onResults(answers, inferenceTime)
}
```
1. Pass the results from `answer` to the listener object.
```
interface AnswererListener {
fun onError(error: String)
fun onResults(
results: List<QaAnswer>?,
inferenceTime: Long
)
}
```
### Handle model output
After you input a question, the model provides a maximum of five possible
answers within the passage.
To get the results from the model:
1. Create an `onResult` function for the listener object to handle the output.
In the example application, the listener object is located in
[BertQaHelper.kt](https://github.com/tensorflow/examples/blob/master/lite/examples/bert_qa/android/app/src/main/java/org/tensorflow/lite/examples/bertqa/BertQaHelper.kt#L92-L98)
```
interface AnswererListener {
fun onError(error: String)
fun onResults(
results: List<QaAnswer>?,
inferenceTime: Long
)
}
```
1. Highlight sections of the passage based on the results. In the example
application, this is located in
[QaFragment.kt](https://github.com/tensorflow/examples/blob/master/lite/examples/bert_qa/android/app/src/main/java/org/tensorflow/lite/examples/bertqa/fragments/QaFragment.kt#L199-L208):
```
override fun onResults(results: List<QaAnswer>?, inferenceTime: Long) {
results?.first()?.let {
highlightAnswer(it.text)
}
fragmentQaBinding.tvInferenceTime.text = String.format(
requireActivity().getString(R.string.bottom_view_inference_time),
inferenceTime
)
}
```
Once the model has returned a set of results, your application can act on those
predictions by presenting the result to your user or executing additional logic.
## Next steps
* Train and implement the models from scratch with the
[Question Answer with TensorFlow Lite Model Maker](https://ai.google.dev/edge/litert/libraries/modify/question_answer)
tutorial.
* Explore more
[text processing tools for TensorFlow](https://www.tensorflow.org/text).
* Download other BERT models on
[TensorFlow Hub](https://tfhub.dev/google/collections/bert/1).
* Explore various uses of TensorFlow Lite in the [examples](../../examples).
tensorflow/lite/g3doc/android/tutorials/text_classification.md
+488
View File
@@ -0,0 +1,488 @@
# Text classification with Android
This tutorial shows you how to build an Android application using TensorFlow
Lite to classify natural language text. This application is designed for a
physical Android device but can also run on a device emulator.
The
[example application](https://github.com/tensorflow/examples/tree/master/lite/examples/text_classification/android)
uses TensorFlow Lite to classify text as either positive or negative, using the
[Task library for natural language (NL)](https://ai.google.dev/edge/litert/libraries/task_library/overview)
to enable execution of the text classification machine learning models.
If you are updating an existing project, you can use the example application as
a reference or template. For instructions on how to add text classification to
an existing application, refer to
[Updating and modifying your application](#modify_applications).
## Text classification overview
*Text classification* is the machine learning task of assigning a set of
predefined categories to open-ended text. A text classification model is trained
on a corpus of natural language text, where words or phrases are manually
classified.
The trained model receives text as input and attempts to categorize the text
according to the set of known classes it was trained to classify. For example,
the models in this example accept a snippet of text and determines whether the
sentiment of the text is positive or negative. For each snippet of text, the
text classification model outputs a score that indicates the confidence of the
text being correctly classified as either positive or negative.
For more information on how the models in this tutorial are generated, refer to
the
[Text classification with TensorFlow Lite Model Maker](https://ai.google.dev/edge/litert/libraries/modify/text_classification)
tutorial.
## Models and dataset
This tutorial uses models that were trained using the
[SST-2](https://nlp.stanford.edu/sentiment/index.html) (Stanford Sentiment
Treebank) dataset. SST-2 contains 67,349 movie reviews for training and 872
movie reviews for testing, with each review categorized as either positive or
negative. The models used in this app were trained using the TensorFlow Lite
[Model Maker](https://ai.google.dev/edge/litert/libraries/modify/text_classification)
tool.
The example application uses the following pre-trained models:
* [Average Word Vector](https://www.tensorflow.org/lite/inference_with_metadata/task_library/nl_classifier)
(`NLClassifier`) - The Task Library's `NLClassifier` classifies input text
into different categories, and can handle most text classification models.
* [MobileBERT](https://www.tensorflow.org/lite/inference_with_metadata/task_library/bert_nl_classifier)
(`BertNLClassifier`) - The Task Library's `BertNLClassifier` is similar to
the NLClassifier but is tailored for cases that require out-of-graph
Wordpiece and Sentencepiece tokenizations.
## Setup and run the example app
To setup the text classification application, download the example app from
[GitHub](https://github.com/tensorflow/examples/tree/master/lite/examples/text_classification/android)
and run it using [Android Studio](https://developer.android.com/studio/).
### System requirements
* **[Android Studio](https://developer.android.com/studio/index.html)**
version 2021.1.1 (Bumblebee) or higher.
* Android SDK version 31 or higher
* Android device with a minimum OS version of SDK 21 (Android 7.0 - Nougat)
with
[developer mode](https://developer.android.com/studio/debug/dev-options)
enabled, or an Android Emulator.
### Get the example code
Create a local copy of the example code. You will use this code to create a
project in Android Studio and run the example application.
To clone and setup the example code:
1. Clone the git repository
<pre class="devsite-click-to-copy">
git clone https://github.com/tensorflow/examples.git
</pre>
2. Optionally, configure your git instance to use sparse checkout, so you have only
the files for the text classification example app:
<pre class="devsite-click-to-copy">
cd examples
git sparse-checkout init --cone
git sparse-checkout set lite/examples/text_classification/android
</pre>
### Import and run the project
Create a project from the downloaded example code, build the project, and then
run it.
To import and build the example code project:
1. Start [Android Studio](https://developer.android.com/studio).
1. From the Android Studio, select **File > New > Import Project**.
1. Navigate to the example code directory containing the build.gradle file
(`.../examples/lite/examples/text_classification/android/build.gradle`) and
select that directory.
1. If Android Studio requests a Gradle Sync, choose OK.
1. Ensure that your Android device is connected to your computer and developer
mode is enabled. Click the green `Run` arrow.
If you select the correct directory, Android Studio creates a new project and
builds it. This process can take a few minutes, depending on the speed of your
computer and if you have used Android Studio for other projects. When the build
completes, the Android Studio displays a `BUILD SUCCESSFUL` message in the
**Build Output** status panel.
To run the project:
1. From Android Studio, run the project by selecting **Run > Run…**.
1. Select an attached Android device (or emulator) to test the app.
### Using the application
![Text classification example app in Android](../../../images/lite/android/text-classification-screenshot.png){: .attempt-right width="250px"}
After running the project in Android Studio, the application will automatically
open on the connected device or device emulator.
To use the text classifier:
1. Enter a snippet of text in the text box.
1. From the **Delegate** drop-down, choose either `CPU` or `NNAPI`.
1. Specify a model by choosing either `AverageWordVec` or `MobileBERT`.
1. Choose **Classify**.
The application outputs a *positive* score and a *negative* score. These two
scores will sum to 1, and measures the likelihood that the sentiment of the
input text is positive or negative. A higher number denotes a higher level of
confidence.
You now have a functioning text classification application. Use the following
sections to better understand how the example application works, and how to
implement text classification features to your production applications:
* [How the application works](#how_it_works) - A walkthrough of the structure
and key files of the example application.
* [Modify your application](#modify_applications) - Instructions on adding
text classification to an existing application.
## How the example app works {:#how_it_works}
The application uses the
[Task library for natural language (NL)](https://ai.google.dev/edge/litert/libraries/task_library/overview)
package to implement the text classification models. The two models, Average
Word Vector and MobileBERT, were trained using the TensorFlow Lite
[Model Maker](https://ai.google.dev/edge/litert/libraries/modify/text_classification).
The application runs on CPU by default, with the option of hardware acceleration
using the NNAPI delegate.
The following files and directories contain the crucial code for this text
classification application:
* [TextClassificationHelper.kt](https://github.com/tensorflow/examples/blob/master/lite/examples/text_classification/android/app/src/main/java/org/tensorflow/lite/examples/textclassification/TextClassificationHelper.kt) -
Initializes the text classifier and handles the model and delegate
selection.
* [MainActivity.kt](https://github.com/tensorflow/examples/blob/master/lite/examples/text_classification/android/app/src/main/java/org/tensorflow/lite/examples/textclassification/MainActivity.kt) -
Implements the application, including calling `TextClassificationHelper` and
`ResultsAdapter`.
* [ResultsAdapter.kt](https://github.com/tensorflow/examples/blob/master/lite/examples/text_classification/android/app/src/main/java/org/tensorflow/lite/examples/textclassification/ResultsAdapter.kt) -
Handles and formats the results.
## Modify your application {:#modify_applications}
The following sections explain the key steps to modify your own Android app to
run the model shown in the example app. These instructions use the example app
as a reference point. The specific changes needed for your own app may vary from
the example app.
### Open or create an Android project
You need an Android development project in Android Studio to follow along with
the rest of these instructions. Follow the instructions below to open an
existing project or create a new one.
To open an existing Android development project:
* In Android Studio, select *File > Open* and select an existing project.
To create a basic Android development project:
* Follow the instructions in Android Studio to
[Create a basic project](https://developer.android.com/studio/projects/create-project).
For more information on using Android Studio, refer to the
[Android Studio documentation](https://developer.android.com/studio/intro).
### Add project dependencies
In your own application, you must add specific project dependencies to run
TensorFlow Lite machine learning models, and access utility functions that
convert data such as strings, into a tensor data format that can be processed by
the model you are using.
The following instructions explain how to add the required project and module
dependencies to your own Android app project.
To add module dependencies:
1. In the module that uses TensorFlow Lite, update the module's `build.gradle`
file to include the following dependencies.
In the example application, the dependencies are located in
[app/build.gradle](https://github.com/tensorflow/examples/blob/master/lite/examples/text_classification/android/app/build.gradle):
```
dependencies {
...
implementation 'org.tensorflow:tensorflow-lite-task-text:0.4.0'
}
```
The project must include the Text task library
(`tensorflow-lite-task-text`).
If you want to modify this app to run on a graphics processing unit (GPU),
the GPU library (`tensorflow-lite-gpu-delegate-plugin`) provides the
infrastructure to run the app on GPU, and Delegate (`tensorflow-lite-gpu`)
provides the compatibility list. Running this app on GPU is outside the
scope of this tutorial.
1. In Android Studio, sync the project dependencies by selecting: **File > Sync
Project with Gradle Files**.
### Initialize the ML models {:#initialize_models}
In your Android app, you must initialize the TensorFlow Lite machine learning
model with parameters before running predictions with the model.
A TensorFlow Lite model is stored as a `*.tflite` file. The model file contains
the prediction logic and typically includes
[metadata](https://ai.google.dev/edge/litert/models/metadata) about how to
interpret prediction results, such as prediction class names. Typically, model
files are stored in the `src/main/assets` directory of your development project,
as in the code example:
- `<project>/src/main/assets/mobilebert.tflite`
- `<project>/src/main/assets/wordvec.tflite`
Note: The example app uses a
`[download_model.gradle](https://github.com/tensorflow/examples/blob/master/lite/examples/text_classification/android/app/download_model.gradle)`
file to download the
[Average Word Vector](https://www.tensorflow.org/lite/inference_with_metadata/task_library/nl_classifier)
and
[MobileBERT](https://www.tensorflow.org/lite/inference_with_metadata/task_library/bert_nl_classifier)
models at build time. This approach is not necessary or recommended for a
production app.
For convenience and code readability, the example declares a companion object
that defines the settings for the model.
To initialize the model in your app:
1. Create a companion object to define the settings for the model. In the
example application, this object is located in
[TextClassificationHelper.kt](https://github.com/tensorflow/examples/blob/master/lite/examples/text_classification/android/app/src/main/java/org/tensorflow/lite/examples/textclassification/TextClassificationHelper.kt):
```
companion object {
const val DELEGATE_CPU = 0
const val DELEGATE_NNAPI = 1
const val WORD_VEC = "wordvec.tflite"
const val MOBILEBERT = "mobilebert.tflite"
}
```
1. Create the settings for the model by building a classifier object, and
construct a TensorFlow Lite object using either `BertNLClassifier` or
`NLClassifier`.
In the example application, this is located in the `initClassifier` function
within
[TextClassificationHelper.kt](https://github.com/tensorflow/examples/blob/master/lite/examples/text_classification/android/app/src/main/java/org/tensorflow/lite/examples/textclassification/TextClassificationHelper.kt):
```
fun initClassifier() {
...
if( currentModel == MOBILEBERT ) {
...
bertClassifier = BertNLClassifier.createFromFileAndOptions(
context,
MOBILEBERT,
options)
} else if (currentModel == WORD_VEC) {
...
nlClassifier = NLClassifier.createFromFileAndOptions(
context,
WORD_VEC,
options)
}
}
```
Note: Most production apps using text classification will use either
`BertNLClassifier` or `NLClassifier` - not both.
### Enable hardware acceleration (optional) {:#hardware_acceleration}
When initializing a TensorFlow Lite model in your app, you should consider using
hardware acceleration features to speed up the prediction calculations of the
model. TensorFlow Lite
[delegates](https://www.tensorflow.org/lite/performance/delegates) are software
modules that accelerate execution of machine learning models using specialized
processing hardware on a mobile device, such as graphics processing unit (GPUs)
or tensor processing units (TPUs).
To enable hardware acceleration in your app:
1. Create a variable to define the delegate that the application will use. In
the example application, this variable is located early in
[TextClassificationHelper.kt](https://github.com/tensorflow/examples/blob/master/lite/examples/text_classification/android/app/src/main/java/org/tensorflow/lite/examples/textclassification/TextClassificationHelper.kt):
```
var currentDelegate: Int = 0
```
1. Create a delegate selector. In the example application, the delegate
selector is located in the `initClassifier` function within
[TextClassificationHelper.kt](https://github.com/tensorflow/examples/blob/master/lite/examples/text_classification/android/app/src/main/java/org/tensorflow/lite/examples/textclassification/TextClassificationHelper.kt):
```
val baseOptionsBuilder = BaseOptions.builder()
when (currentDelegate) {
DELEGATE_CPU -> {
// Default
}
DELEGATE_NNAPI -> {
baseOptionsBuilder.useNnapi()
}
}
```
Note: It is possible to modify this app to use a GPU delegate, but this requires
that the classifier be created on the same thread that is using the classifier.
This is outside of the scope of this tutorial.
Using delegates for running TensorFlow Lite models is recommended, but not
required. For more information about using delegates with TensorFlow Lite, see
[TensorFlow Lite Delegates](https://www.tensorflow.org/lite/performance/delegates).
### Prepare data for the model
In your Android app, your code provides data to the model for interpretation by
transforming existing data such as raw text into a
[Tensor](https://www.tensorflow.org/lite/api_docs/java/org/tensorflow/lite/Tensor)
data format that can be processed by your model. The data in a Tensor you pass
to a model must have specific dimensions, or shape, that matches the format of
data used to train the model.
This text classification app accepts a
[string](https://developer.android.com/reference/java/lang/String.html) as
input, and the models are trained exclusively on an English language corpus.
Special characters and non-English words are ignored during inference.
To provide text data to the model:
1. Ensure that the `initClassifier` function contains the code for the delegate
and models, as explained in the
[Initialize the ML models](#initialize_models) and
[Enable hardware acceleration](#hardware_acceleration) sections.
1. Use the `init` block to call the `initClassifier` function. In the example
application, the `init` is located in
[TextClassificationHelper.kt](https://github.com/tensorflow/examples/blob/master/lite/examples/text_classification/android/app/src/main/java/org/tensorflow/lite/examples/textclassification/TextClassificationHelper.kt):
```
init {
initClassifier()
}
```
### Run predictions
In your Android app, once you have initialized either a
[BertNLClassifier](https://www.tensorflow.org/lite/api_docs/java/org/tensorflow/lite/task/text/nlclassifier/BertNLClassifier)
or
[NLClassifier](https://www.tensorflow.org/lite/api_docs/java/org/tensorflow/lite/task/text/nlclassifier/NLClassifier)
object, you can begin feeding input text for the model to categorize as
"positive" or "negative".
To run predictions:
1. Create a `classify` function, which uses the selected classifier
(`currentModel`) and measures the time taken to classify the input text
(`inferenceTime`). In the example application, the `classify` function is
located in
[TextClassificationHelper.kt](https://github.com/tensorflow/examples/blob/master/lite/examples/text_classification/android/app/src/main/java/org/tensorflow/lite/examples/textclassification/TextClassificationHelper.kt):
```
fun classify(text: String) {
executor = ScheduledThreadPoolExecutor(1)
executor.execute {
val results: List<Category>
// inferenceTime is the amount of time, in milliseconds, that it takes to
// classify the input text.
var inferenceTime = SystemClock.uptimeMillis()
// Use the appropriate classifier based on the selected model
if(currentModel == MOBILEBERT) {
results = bertClassifier.classify(text)
} else {
results = nlClassifier.classify(text)
}
inferenceTime = SystemClock.uptimeMillis() - inferenceTime
listener.onResult(results, inferenceTime)
}
}
```
1. Pass the results from `classify` to the listener object.
```
fun classify(text: String) {
...
listener.onResult(results, inferenceTime)
}
```
### Handle model output
After you input a line of text, the model produces a prediction score, expressed
as a Float, between 0 and 1 for the 'positive' and 'negative' categories.
To get the prediction results from the model:
1. Create an `onResult` function for the listener object to handle the output.
In the example application, the listener object is located in
[MainActivity.kt](https://github.com/tensorflow/examples/blob/master/lite/examples/text_classification/android/app/src/main/java/org/tensorflow/lite/examples/textclassification/MainActivity.kt)
```
private val listener = object : TextClassificationHelper.TextResultsListener {
override fun onResult(results: List<Category>, inferenceTime: Long) {
runOnUiThread {
activityMainBinding.bottomSheetLayout.inferenceTimeVal.text =
String.format("%d ms", inferenceTime)
adapter.resultsList = results.sortedByDescending {
it.score
}
adapter.notifyDataSetChanged()
}
}
...
}
```
1. Add an `onError` function to the listener object to handle errors:
```
private val listener = object : TextClassificationHelper.TextResultsListener {
...
override fun onError(error: String) {
Toast.makeText(this@MainActivity, error, Toast.LENGTH_SHORT).show()
}
}
```
Once the model has returned a set of prediction results, your application can
act on those predictions by presenting the result to your user or executing
additional logic. The example application lists the prediction scores in the
user interface.
## Next steps
* Train and implement the models from scratch with the
[Text classification with TensorFlow Lite Model Maker](https://ai.google.dev/edge/litert/libraries/modify/text_classification)
tutorial.
* Explore more
[text processing tools for TensorFlow](https://www.tensorflow.org/text).
* Download other BERT models on
[TensorFlow Hub](https://tfhub.dev/google/collections/bert/1).
* Explore various uses of TensorFlow Lite in the [examples](../../examples).
* Learn more about using machine learning models with TensorFlow Lite in the
[Models](../../models) section.
* Learn more about implementing machine learning in your mobile application in
the [TensorFlow Lite Developer Guide](../../guide).