chore: import upstream snapshot with attribution

tensorflow/lite/g3doc/examples/audio_classification/overview.md

@@ -0,0 +1,139 @@
+# Audio classification
+
+<img src="../images/audio.png" class="attempt-right">
+
+The task of identifying what an audio represents is called _audio
+classification_. An audio classification model is trained to recognize various
+audio events. For example, you may train a model to recognize events
+representing three different events: clapping, finger snapping, and typing.
+TensorFlow Lite provides optimized pre-trained models that you can deploy in
+your mobile applications. Learn more about audio classification using TensorFlow
+[here](https://www.tensorflow.org/tutorials/audio/simple_audio).
+
+The following image shows the output of the audio classification model on
+Android.
+
+<img src="images/android_audio_classification.png" alt="Screenshot of Android example" width="30%">
+
+Note: (1) To integrate an existing model, try
+[TensorFlow Lite Task Library](https://www.tensorflow.org/lite/inference_with_metadata/task_library/audio_classifier).
+(2) To customize a model, try
+[TensorFlow Lite Model Maker](https://ai.google.dev/edge/litert/libraries/modify/audio_classification).
+
+## Get started
+
+If you are new to TensorFlow Lite and are working with Android, we recommend
+exploring the following example applications that can help you get started.
+
+You can leverage the out-of-box API from
+[TensorFlow Lite Task Library](../../inference_with_metadata/task_library/audio_classifier.md)
+to integrate audio classification models in just a few lines of code. You can
+also build your own custom inference pipeline using the
+[TensorFlow Lite Support Library](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/inference_with_metadata/lite_support.md).
+
+The Android example below demonstrates the implementation using the
+[TFLite Task Library](https://github.com/tensorflow/examples/tree/master/lite/examples/audio_classification/android)
+
+<a class="button button-primary" href="https://github.com/tensorflow/examples/tree/master/lite/examples/audio_classification/android">View
+Android example</a>
+
+<a class="button button-primary" href="https://github.com/tensorflow/examples/tree/master/lite/examples/sound_classification/ios">View
+iOS example</a>
+
+If you are using a platform other than Android/iOS, or if you are already
+familiar with the
+[TensorFlow Lite APIs](https://www.tensorflow.org/api_docs/python/tf/lite),
+download the starter model and supporting files (if applicable).
+
+<a class="button button-primary" href="https://tfhub.dev/google/lite-model/yamnet/classification/tflite/1?lite-format=tflite">Download
+starter model from TensorFlow Hub</a>
+
+## Model description
+
+YAMNet is an audio event classifier that takes audio waveform as input and makes
+independent predictions for each of 521 audio events from the
+[AudioSet](https://g.co/audioset) ontology. The model uses the MobileNet v1
+architecture and was trained using the AudioSet corpus. This model was
+originally released in the TensorFlow Model Garden, where is the model source
+code, the original model checkpoint, and more detailed documentation.
+
+### How it works
+
+There are two versions of the YAMNet model converted to TFLite:
+
+*   [YAMNet](https://tfhub.dev/google/yamnet/1) Is the original audio
+    classification model, with dynamic input size, suitable for Transfer
+    Learning, Web and Mobile deployment. It also has a more complex output.
+
+*   [YAMNet/classification](https://tfhub.dev/google/lite-model/yamnet/classification/tflite/1)
+    is a quantized version with a simpler fixed length frame input (15600
+    samples) and return a single vector of scores for 521 audio event classes.
+
+### Inputs
+
+The model accepts a 1-D `float32` Tensor or NumPy array of length 15600
+containing a 0.975 second waveform represented as mono 16 kHz samples in the
+range `[-1.0, +1.0]`.
+
+### Outputs
+
+The model returns a 2-D `float32` Tensor of shape (1, 521) containing the
+predicted scores for each of the 521 classes in the AudioSet ontology that are
+supported by YAMNet. The column index (0-520) of the scores tensor is mapped to
+the corresponding AudioSet class name using the YAMNet Class Map, which is
+available as an associated file `yamnet_label_list.txt` packed into the model
+file. See below for usage.
+
+### Suitable uses
+
+YAMNet can be used
+
+*   as a stand-alone audio event classifier that provides a reasonable baseline
+    across a wide variety of audio events.
+*   as a high-level feature extractor: the 1024-D embedding output of YAMNet can
+    be used as the input features of another model which can then be trained on
+    a small amount of data for a particular task. This allows quickly creating
+    specialized audio classifiers without requiring a lot of labeled data and
+    without having to train a large model end-to-end.
+*   as a warm start: the YAMNet model parameters can be used to initialize part
+    of a larger model which allows faster fine-tuning and model exploration.
+
+### Limitations
+
+*   YAMNet's classifier outputs have not been calibrated across classes, so you
+    cannot directly treat the outputs as probabilities. For any given task, you
+    will very likely need to perform a calibration with task-specific data which
+    lets you assign proper per-class score thresholds and scaling.
+*   YAMNet has been trained on millions of YouTube videos and although these are
+    very diverse, there can still be a domain mismatch between the average
+    YouTube video and the audio inputs expected for any given task. You should
+    expect to do some amount of fine-tuning and calibration to make YAMNet
+    usable in any system that you build.
+
+## Model customization
+
+The pre-trained models provided are trained to detect 521 different audio
+classes. For a full list of classes, see the labels file in the
+<a href="https://github.com/tensorflow/models/blob/master/research/audioset/yamnet/yamnet_class_map.csv">model
+repository</a>.
+
+You can use a technique known as transfer learning to re-train a model to
+recognize classes not in the original set. For example, you could re-train the
+model to detect multiple bird songs. To do this, you will need a set of training
+audios for each of the new labels you wish to train. The recommended way is to
+use
+[TensorFlow Lite Model Maker](https://ai.google.dev/edge/litert/libraries/modify/audio_classification)
+library which simplifies the process of training a TensorFlow Lite model using
+custom dataset, in a few lines of codes. It uses transfer learning to reduce the
+amount of required training data and time. You can also learn from
+[Transfer learning for audio recognition](https://www.tensorflow.org/tutorials/audio/transfer_learning_audio)
+as an example of transfer learning.
+
+## Further reading and resources
+
+Use the following resources to learn more about concepts related to audio
+classification:
+
+*   [Audio classification using TensorFlow](https://www.tensorflow.org/tutorials/audio/simple_audio)
+*   [Transfer learning for audio recognition](https://www.tensorflow.org/tutorials/audio/transfer_learning_audio)
+*   [Audio data augmentation](https://www.tensorflow.org/io/tutorials/audio)

tensorflow/lite/g3doc/examples/auto_complete/overview.md

@@ -0,0 +1,340 @@
+# Auto Complete
+
+<table class="tfo-notebook-buttons" align="left">
+  <td>
+    <a target="_blank" href="https://www.tensorflow.org/lite/examples/auto_complete/overview"><img src="https://www.tensorflow.org/images/tf_logo_32px.png" />View on TensorFlow.org</a>
+  </td>
+  <td>
+    <a target="_blank" href="https://colab.sandbox.google.com/github/tensorflow/codelabs/blob/main/KerasNLP/io2023_workshop.ipynb"><img src="https://www.tensorflow.org/images/colab_logo_32px.png" />Run in Google Colab</a>
+  </td>
+</table>
+
+## Introduction
+
+Large language models (LLMs) are a class of machine learning models that are
+trained to generate text based on large datasets. They can be used for natural
+language processing (NLP) tasks, including text generation, question answering,
+and machine translation. They are based on Transformer architecture and are
+trained on massive amounts of text data, often involving billions of words. Even
+LLMs of a smaller scale, such as GPT-2, can perform impressively. Converting
+TensorFlow models to a lighter, faster, and low-power model allows for us to run
+generative AI models on-device, with benefits of better user security because
+data will never leave your device.
+
+This runbook shows you how to build an Android app with TensorFlow Lite to run a
+Keras LLM and provides suggestions for model optimization using quantizing
+techniques, which otherwise would require a much larger amount of memory and
+greater computational power to run.
+
+We have open sourced our
+[Android app framework](https://github.com/tensorflow/examples/tree/master/lite/examples/generative_ai/)
+that any compatible TFLite LLMs can plug into. Here are two demos:
+
+*   In Figure 1, we used a Keras GPT-2 model to perform text completion tasks on
+    device.
+*   In Figure 2, we converted a version of instruction-tuned
+    [PaLM model](https://ai.googleblog.com/2022/04/pathways-language-model-palm-scaling-to.html)
+    (1.5 billion parameters) to TFLite and executed through TFLite runtime.
+
+<center>
+![Autocomplete with PaLM](https://storage.googleapis.com/download.tensorflow.org/tflite/examples/autocomplete_fig1.gif){: width="400px"}
+<figcaption><b>Figure 1: </b>Example of running the Keras GPT-2 model (converted
+from this [Codelab](https://codelabs.developers.google.com/kerasnlp-tflite)) on
+device to perform text completion on Pixel 7. Demo shows the real latency with
+no speedup.</figcaption>
+</center>
+
+<center>
+![Autocomplete with PaLM](https://storage.googleapis.com/download.tensorflow.org/tflite/examples/autocomplete_fig2.gif){: width="400px"}
+</p>
+<figcaption><b>Figure 2: </b>Example of running a version of
+[PaLM model](https://ai.googleblog.com/2022/04/pathways-language-model-palm-scaling-to.html)
+with 1.5 billion parameters. Demo is recorded on Pixel 7 Pro without playback
+speedup.</figcaption>
+</center>
+
+## Guides
+
+### Model authoring
+
+For this demonstration, we will use KerasNLP to get the GPT-2 model. KerasNLP is
+a library that contains state-of-the-art pretrained models for natural language
+processing tasks, and can support users through their entire development cycle.
+You can see the list of models available in the
+[KerasNLP repository](https://keras.io/api/keras_hub/models/). The workflows are
+built from modular components that have state-of-the-art preset weights and
+architectures when used out-of-the-box and are easily customizable when more
+control is needed. Creating the GPT-2 model can be done with the following
+steps:
+
+```python
+gpt2_tokenizer = keras_nlp.models.GPT2Tokenizer.from_preset("gpt2_base_en")
+
+gpt2_preprocessor = keras_nlp.models.GPT2CausalLMPreprocessor.from_preset(
+    "gpt2_base_en",
+    sequence_length=256,
+    add_end_token=True,
+)
+
+gpt2_lm =
+keras_nlp.models.GPT2CausalLM.from_preset(
+"gpt2_base_en",
+preprocessor=gpt2_preprocessor
+)
+```
+
+One commonality among these three lines of code is the `from_preset()` method,
+which will instantiate the part of Keras API from a preset architecture and/or
+weights, therefore loading the pre-trained model. From this code snippet, you’ll
+also notice three modular components:
+
+1.  **Tokenizer**: converts a raw string input into integer token IDs suitable
+    for a Keras Embedding layer. GPT-2 uses the byte-pair encoding (BPE)
+    tokenizer specifically.
+
+2.  **Preprocessor**: layer for tokenizing and packing inputs to be fed into a
+    Keras model. Here, the preprocessor will pad the tensor of token IDs to a
+    specified length (256) after tokenization.
+
+3.  **Backbone**: Keras model that follows the SoTA transformer backbone
+    architecture and has the preset weights.
+
+Additionally, you can check out the full GPT-2 model implementation on
+[GitHub](https://github.com/keras-team/keras-nlp/tree/master/keras_nlp/models/gpt2).
+
+### Model conversion
+
+TensorFlow Lite is a mobile library for deploying methods on mobile,
+microcontrollers, and other edge devices. The first step is to convert a Keras
+model to a more compact TensorFlow Lite format using the TensorFlow Lite
+**converter**, and then use the TensorFlow Lite **interpreter**, which is highly
+optimized for mobile devices, to run the converted model.
+
+<img src="https://www.tensorflow.org/lite/examples/auto_complete/images/tflite_workflow.png" class="attempt-right" />
+Start with the `generate()` function from `GPT2CausalLM` that performs the
+conversion. Wrap the `generate()` function to create a concrete TensorFlow
+function:
+
+```python
+@tf.function
+def generate(prompt, max_length):
+    """
+    Args:
+        prompt: input prompt to the LLM in string format
+        max_length: the max length of the generated tokens
+    """
+    return gpt2_lm.generate(prompt, max_length)
+
+concrete_func = generate.get_concrete_function(tf.TensorSpec([], tf.string), 100)
+```
+
+Note that you can also use `from_keras_model()` from
+[`TFLiteConverter`](https://www.tensorflow.org/api_docs/python/tf/lite/TFLiteConverter#from_keras_model)
+in order to perform the conversion.
+
+Now define a helper function that will run inference with an input and a TFLite
+model. TensorFlow text ops are not built-in ops in the TFLite runtime, so you
+will need to add these custom ops in order for the interpreter to make inference
+on this model. This helper function accepts an input and a function that
+performs the conversion, namely the `generator()` function defined above.
+
+```python
+def run_inference(input, generate_tflite):
+    interp = interpreter.InterpreterWithCustomOps(
+        model_content=generate_tflite,
+        custom_op_registerers=
+            tf_text.tflite_registrar.SELECT_TFTEXT_OPS
+    )
+
+    interp.get_signature_list()
+
+    generator = interp.get_signature_runner('serving_default')
+    output = generator(prompt=np.array([input]))
+```
+
+You can convert the model now:
+
+```python
+gpt2_lm.jit_compile = False
+converter = tf.lite.TFLiteConverter.from_concrete_functions(
+    [concrete_func],
+    gpt2_lm)
+
+converter.target_spec.supported_ops = [
+    tf.lite.OpsSet.TFLITE_BUILTINS, # enable TFLite ops
+    tf.lite.OpsSet.SELECT_TF_OPS, # enable TF ops
+]
+converter.allow_custom_ops = True
+converter.target_spec.experimental_select_user_tf_ops = [
+    "UnsortedSegmentJoin",
+    "UpperBound"
+]
+converter._experimental_guarantee_all_funcs_one_use = True
+generate_tflite = converter.convert()
+run_inference("I'm enjoying a", generate_tflite)
+```
+
+### Quantization
+
+TensorFlow Lite has implemented an optimization technique called
+**quantization** which can reduce model size and accelerate inference. Through
+the quantization process, 32-bit floats are mapped to smaller 8-bit integers,
+therefore reducing the model size by a factor of 4 for more efficient execution
+on modern hardwares. There are several ways to do quantization in TensorFlow.
+You can visit the
+[TFLite Model optimization](https://www.tensorflow.org/lite/performance/model_optimization)
+and
+[TensorFlow Model Optimization Toolkit](https://www.tensorflow.org/model_optimization)
+pages for more information. The types of quantizations are explained briefly
+below.
+
+Here, you will use the
+[post-training dynamic range quantization](https://www.tensorflow.org/lite/performance/post_training_quant)
+on the GPT-2 model by setting the converter optimization flag to
+`tf.lite.Optimize.DEFAULT`, and the rest of the conversion process is the same
+as detailed before. We tested that with this quantization technique the latency
+is around 6.7 seconds on Pixel 7 with max output length set to 100.
+
+```python
+gpt2_lm.jit_compile = False
+converter = tf.lite.TFLiteConverter.from_concrete_functions(
+    [concrete_func],
+    gpt2_lm)
+
+converter.target_spec.supported_ops = [
+    tf.lite.OpsSet.TFLITE_BUILTINS, # enable TFLite ops
+    tf.lite.OpsSet.SELECT_TF_OPS, # enable TF ops
+]
+converter.allow_custom_ops = True
+converter.optimizations = [tf.lite.Optimize.DEFAULT]
+converter.target_spec.experimental_select_user_tf_ops = [
+    "UnsortedSegmentJoin",
+    "UpperBound"
+]
+converter._experimental_guarantee_all_funcs_one_use = True
+quant_generate_tflite = converter.convert()
+run_inference("I'm enjoying a", quant_generate_tflite)
+```
+
+**Dynamic Range**
+
+Dynamic range quantization is the recommended starting point for optimizing
+on-device models. It can achieve about a 4x reduction in the model size, and is
+a recommended starting point as it provides reduced memory usage and faster
+computation without you having to provide a representative dataset for
+calibration. This type of quantization statically quantizes only the weights
+from floating point to 8-bit integer at conversion time.
+
+**FP16**
+
+Floating point models can also be optimized by quantizing the weights to float16
+type. The advantages of
+[float16 quantization](https://www.tensorflow.org/lite/performance/post_training_float16_quant)
+are reducing the model size by up to half (as all weights become half their
+size), causing minimal loss in accuracy, and supporting GPU delegates that can
+operate directly on float16 data (which results in faster computation than on
+float32 data). A model converted to float16 weights can still run on the CPU
+without additional modifications. The float16 weights are upsampled to float32
+before the first inference, which permits a reduction in model size in exchange
+for a minimal impact to latency and accuracy.
+
+**Full Integer Quantization**
+
+[Full integer quantization](https://www.tensorflow.org/lite/performance/post_training_integer_quant)
+both converts the 32 bit floating point numbers, including weights and
+activations, to the nearest 8 bit integers. This type of quantization results in
+a smaller model with increased inference speed, which is incredibly valuable
+when using microcontrollers. This mode is recommended when activations are
+sensitive to the quantization.
+
+### Android App integration
+
+You can follow this
+[Android example](https://github.com/tensorflow/examples/tree/master/lite/examples/generative_ai)
+to integrate your TFLite model into an Android App.
+
+### Prerequisites
+
+If you have not already, install
+[Android Studio](https://developer.android.com/studio/index.html), following the
+instructions on the website.
+
+*   Android Studio 2022.2.1 or above.
+*   An Android device or Android emulator with more than 4G memory
+
+### Building and Running with Android Studio
+
+*   Open Android Studio, and from the Welcome screen, select **Open an existing
+    Android Studio project**.
+*   From the Open File or Project window that appears, navigate to and select
+    the
+    [`lite/examples/generative_ai/android`](https://github.com/tensorflow/examples/tree/master/lite/examples/generative_ai/android)
+    directory from wherever you cloned the TensorFlow Lite sample GitHub repo.
+*   You may also need to install various platforms and tools according to error
+    messages.
+*   Rename the converted .tflite model to `autocomplete.tflite` and copy it into
+    `app/src/main/assets/` folder.
+*   Select menu **Build -> Make Project** to build the app. (Ctrl+F9, depending
+    on your version).
+*   Click menu **Run -> Run 'app'**. (Shift+F10, depending on your version)
+
+Alternatively, you can also use the
+[gradle wrapper](https://docs.gradle.org/current/userguide/gradle_wrapper.html#gradle_wrapper)
+to build it in the command line. Please refer to the
+[Gradle documentation](https://docs.gradle.org/current/userguide/command_line_interface.html)
+for more information.
+
+### (Optional) Building the .aar file
+
+By default the app automatically downloads the needed `.aar` files. But if you
+want to build your own, switch to `app/libs/build_aar/` folder run
+`./build_aar.sh`. This script will pull in the necessary ops from TensorFlow
+Text and build the aar for Select TF operators.
+
+After compilation, a new file `tftext_tflite_flex.aar` is generated. Replace the
+.aar file in `app/libs/` folder and re-build the app.
+
+Note that you still need to include the standard `tensorflow-lite` aar in your
+gradle file.
+
+### Context window size
+
+<img src="https://www.tensorflow.org/lite/examples/auto_complete/images/context_window.png" class="attempt-right" />
+
+The app has a changeable parameter ‘context window size’, which is needed
+because LLMs today generally have a fixed context size which limits how many
+words/tokens can be fed into the model as ‘prompt’ (note that ‘word’ is not
+necessarily equivalent to ‘token’ in this case, due to different tokenization
+methods). This number is important because:
+
+*   Setting it too small, the model will not have enough context to generate
+    meaningful output
+*   Setting it too big, the model will not have enough room to work with (since
+    the output sequence is inclusive of the prompt)
+
+You can experiment with it, but setting it to ~50% of output sequence length is
+a good start.
+
+## Safety and Responsible AI
+
+As noted in the original
+[OpenAI GPT-2 announcement](https://openai.com/research/better-language-models),
+there are
+[notable caveats and limitations](https://github.com/openai/gpt-2#some-caveats)
+with the GPT-2 model. In fact, LLMs today generally have some well-known
+challenges such as hallucinations, fairness, and bias; this is because these
+models are trained on real-world data, which make them reflect real world
+issues.
+
+This codelab is created only to demonstrate how to create an app powered by LLMs
+with TensorFlow tooling. The model produced in this codelab is for educational
+purposes only and not intended for production usage.
+
+LLM production usage requires thoughtful selection of training datasets and
+comprehensive safety mitigations. One such functionality offered in this Android
+app is the profanity filter, which rejects bad user inputs or model outputs. If
+any inappropriate language is detected, the app will in return reject that
+action. To learn more about Responsible AI in the context of LLMs, make sure to
+watch the Safe and Responsible Development with Generative Language Models
+technical session at Google I/O 2023 and check out the
+[Responsible AI Toolkit](https://www.tensorflow.org/responsible_ai).

tensorflow/lite/g3doc/examples/bert_qa/overview.md

@@ -0,0 +1,134 @@
+# BERT Question and Answer
+
+Use a TensorFlow Lite model to answer questions based on the content of a given
+passage.
+
+Note: (1) To integrate an existing model, try
+[TensorFlow Lite Task Library](https://www.tensorflow.org/lite/inference_with_metadata/task_library/bert_question_answerer).
+(2) To customize a model, try
+[TensorFlow Lite Model Maker](https://ai.google.dev/edge/litert/libraries/modify/question_answer).
+
+## Get started
+
+<img src="images/screenshot.gif" class="attempt-right" style="max-width: 300px">
+
+If you are new to TensorFlow Lite and are working with Android or iOS, we
+recommend exploring the following example applications that can help you get
+started.
+
+<a class="button button-primary" href="https://github.com/tensorflow/examples/tree/master/lite/examples/bert_qa/android">Android
+example</a>
+<a class="button button-primary" href="https://github.com/tensorflow/examples/tree/master/lite/examples/bert_qa/ios">iOS
+example</a>
+
+If you are using a platform other than Android/iOS, or you are already familiar
+with the
+[TensorFlow Lite APIs](https://www.tensorflow.org/api_docs/python/tf/lite), you
+can download our starter question and answer model.
+
+<a class="button button-primary" href="https://tfhub.dev/tensorflow/lite-model/mobilebert/1/metadata/1?lite-format=tflite">Download
+starter model and vocab</a>
+
+For more information about metadata and associated fields (e.g. `vocab.txt`) see
+<a href="https://www.tensorflow.org/lite/models/convert/metadata#read_the_metadata_from_models">Read
+the metadata from models</a>.
+
+## How it works
+
+The model can be used to build a system that can answer users’ questions in
+natural language. It was created using a pre-trained BERT model fine-tuned on
+SQuAD 1.1 dataset.
+
+[BERT](https://github.com/google-research/bert), or Bidirectional Encoder
+Representations from Transformers, is a method of pre-training language
+representations which obtains state-of-the-art results on a wide array of
+Natural Language Processing tasks.
+
+This app uses a compressed version of BERT, MobileBERT, that runs 4x faster and
+has 4x smaller model size.
+
+[SQuAD](https://rajpurkar.github.io/SQuAD-explorer/), or Stanford Question
+Answering Dataset, is a reading comprehension dataset consisting of articles
+from Wikipedia and a set of question-answer pairs for each article.
+
+The model takes a passage and a question as input, then returns a segment of the
+passage that most likely answers the question. It requires semi-complex
+pre-processing including tokenization and post-processing steps that are
+described in the BERT [paper](https://arxiv.org/abs/1810.04805) and implemented
+in the sample app.
+
+## Performance benchmarks
+
+Performance benchmark numbers are generated with the tool
+[described here](https://www.tensorflow.org/lite/performance/benchmarks).
+
+<table>
+  <thead>
+    <tr>
+      <th>Model Name</th>
+      <th>Model size </th>
+      <th>Device </th>
+      <th>CPU</th>
+    </tr>
+  </thead>
+  <tr>
+    <td rowspan = 3>
+      <a href="https://tfhub.dev/tensorflow/lite-model/mobilebert/1/metadata/1?lite-format=tflite">Mobile Bert</a>
+    </td>
+    <td rowspan = 3>
+      100.5 Mb
+    </td>
+    <td>Pixel 3 (Android 10) </td>
+    <td>123ms*</td>
+  </tr>
+   <tr>
+     <td>Pixel 4 (Android 10) </td>
+    <td>74ms*</td>
+  </tr>
+   <tr>
+     <td>iPhone XS (iOS 12.4.1) </td>
+    <td>257ms** </td>
+  </tr>
+</table>
+
+\* 4 threads used.
+
+\*\* 2 threads used on iPhone for the best performance result.
+
+## Example output
+
+### Passage (Input)
+
+> Google LLC is an American multinational technology company that specializes in
+> Internet-related services and products, which include online advertising
+> technologies, search engine, cloud computing, software, and hardware. It is
+> considered one of the Big Four technology companies, alongside Amazon, Apple,
+> and Facebook.
+>
+> Google was founded in September 1998 by Larry Page and Sergey Brin while they
+> were Ph.D. students at Stanford University in California. Together they own
+> about 14 percent of its shares and control 56 percent of the stockholder
+> voting power through supervoting stock. They incorporated Google as a
+> California privately held company on September 4, 1998, in California. Google
+> was then reincorporated in Delaware on October 22, 2002. An initial public
+> offering (IPO) took place on August 19, 2004, and Google moved to its
+> headquarters in Mountain View, California, nicknamed the Googleplex. In August
+> 2015, Google announced plans to reorganize its various interests as a
+> conglomerate called Alphabet Inc. Google is Alphabet's leading subsidiary and
+> will continue to be the umbrella company for Alphabet's Internet interests.
+> Sundar Pichai was appointed CEO of Google, replacing Larry Page who became the
+> CEO of Alphabet.
+
+### Question (Input)
+
+> Who is the CEO of Google?
+
+### Answer (Output)
+
+> Sundar Pichai
+
+## Read more about BERT
+
+*   Academic paper: [BERT: Pre-training of Deep Bidirectional Transformers for
+    Language Understanding](https://arxiv.org/abs/1810.04805)
+*   [Open-source implementation of BERT](https://github.com/google-research/bert)

tensorflow/lite/g3doc/examples/image_classification/overview.md

@@ -0,0 +1,291 @@
+# Image classification
+
+<img src="../images/image.png" class="attempt-right">
+
+The task of identifying what an image represents is called _image
+classification_. An image classification model is trained to recognize various
+classes of images. For example, you may train a model to recognize photos
+representing three different types of animals: rabbits, hamsters, and dogs.
+TensorFlow Lite provides optimized pre-trained models that you can deploy in
+your mobile applications. Learn more about image classification using TensorFlow
+[here](https://www.tensorflow.org/tutorials/images/classification).
+
+The following image shows the output of the image classification model on
+Android.
+
+<img src="images/android_banana.png" alt="Screenshot of Android example" width="30%">
+
+Note: (1) To integrate an existing model, try
+[TensorFlow Lite Task Library](https://www.tensorflow.org/lite/inference_with_metadata/task_library/image_classifier).
+(2) To customize a model, try
+[TensorFlow Lite Model Maker](https://www.tensorflow.org/lite/models/modify/model_maker/image_classification).
+
+## Get started
+
+If you are new to TensorFlow Lite and are working with Android or iOS, it is
+recommended you explore the following example applications that can help you get
+started.
+
+You can leverage the out-of-box API from
+[TensorFlow Lite Task Library](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/inference_with_metadata/task_library/image_classifier.md)
+to integrate image classification models in just a few lines of code. You can
+also build your own custom inference pipeline using the
+[TensorFlow Lite Support Library](../../inference_with_metadata/lite_support).
+
+The Android example below demonstrates the implementation for both methods as
+[lib_task_api](https://github.com/tensorflow/examples/tree/master/lite/examples/image_classification/android/lib_task_api)
+and
+[lib_support](https://github.com/tensorflow/examples/tree/master/lite/examples/image_classification/android/lib_support),
+respectively.
+
+<a class="button button-primary" href="https://github.com/tensorflow/examples/tree/master/lite/examples/image_classification/android">View
+Android example</a>
+
+<a class="button button-primary" href="https://github.com/tensorflow/examples/tree/master/lite/examples/image_classification/ios">View
+iOS example</a>
+
+If you are using a platform other than Android/iOS, or if you are already
+familiar with the
+[TensorFlow Lite APIs](https://www.tensorflow.org/api_docs/python/tf/lite),
+download the starter model and supporting files (if applicable).
+
+<a class="button button-primary" href="https://storage.googleapis.com/download.tensorflow.org/models/tflite/mobilenet_v1_1.0_224_quant_and_labels.zip">Download
+starter model</a>
+
+## Model description
+
+### How it works
+
+During training, an image classification model is fed images and their
+associated _labels_. Each label is the name of a distinct concept, or class,
+that the model will learn to recognize.
+
+Given sufficient training data (often hundreds or thousands of images per
+label), an image classification model can learn to predict whether new images
+belong to any of the classes it has been trained on. This process of prediction
+is called _inference_. Note that you can also use
+[transfer learning](https://www.tensorflow.org/tutorials/images/transfer_learning)
+to identify new classes of images by using a pre-existing model. Transfer
+learning does not require a very large training dataset.
+
+When you subsequently provide a new image as input to the model, it will output
+the probabilities of the image representing each of the types of animal it was
+trained on. An example output might be as follows:
+
+<table style="width: 40%;">
+  <thead>
+    <tr>
+      <th>Animal type</th>
+      <th>Probability</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td>Rabbit</td>
+      <td>0.07</td>
+    </tr>
+    <tr>
+      <td>Hamster</td>
+      <td>0.02</td>
+    </tr>
+    <tr>
+      <td style="background-color: #fcb66d;">Dog</td>
+      <td style="background-color: #fcb66d;">0.91</td>
+    </tr>
+  </tbody>
+</table>
+
+Each number in the output corresponds to a label in the training data.
+Associating the output with the three labels the model was trained on, you can
+see that the model has predicted a high probability that the image represents a
+dog.
+
+You might notice that the sum of all the probabilities (for rabbit, hamster, and
+dog) is equal to 1. This is a common type of output for models with multiple
+classes (see
+<a href="https://developers.google.com/machine-learning/crash-course/multi-class-neural-networks/softmax">Softmax</a>
+for more information).
+
+Note: Image classification can only tell you the probability that an image
+represents one or more of the classes that the model was trained on. It cannot
+tell you the position or identity of objects within the image. If you need to
+identify objects and their positions within images, you should use an
+<a href="../object_detection/overview">object detection</a> model.
+
+<h4>Ambiguous results</h4>
+
+Since the output probabilities will always sum to 1, if an image is not
+confidently recognized as belonging to any of the classes the model was trained
+on you may see the probability distributed throughout the labels without any one
+value being significantly larger.
+
+For example, the following might indicate an ambiguous result:
+
+<table style="width: 40%;">
+  <thead>
+    <tr>
+      <th>Label</th>
+      <th>Probability</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td>rabbit</td>
+      <td>0.31</td>
+    </tr>
+    <tr>
+      <td>hamster</td>
+      <td>0.35</td>
+    </tr>
+    <tr>
+      <td>dog</td>
+      <td>0.34</td>
+    </tr>
+  </tbody>
+</table>
+If your model frequently returns ambiguous results, you may need a different,
+more accurate model.
+
+<h3>Choosing a model architecture</h3>
+
+TensorFlow Lite provides you with a variety of image classification models which
+are all trained on the original dataset. Model architectures like MobileNet,
+Inception, and NASNet are available on
+<a href="https://tfhub.dev/s?deployment-format=lite">TensorFlow Hub</a>. To
+choose the best model for
+your use case, you need to consider the individual architectures as well as some
+of the tradeoffs between various models. Some of these model tradeoffs are based
+on metrics such as performance, accuracy, and model size. For example, you might
+need a faster model for building a bar code scanner while you might prefer a
+slower, more accurate model for a medical imaging app.
+
+Note that the <a href=https://www.tensorflow.org/lite/guide/hosted_models#image_classification>image classification models</a> provided accept varying sizes of input. For some models, this is indicated in the filename. For example, the Mobilenet_V1_1.0_224 model accepts an input of 224x224 pixels. All of the models require three color channels per pixel (red, green, and blue). Quantized models require 1 byte per channel, and float models require 4 bytes per channel. The <a href="https://github.com/tensorflow/examples/tree/master/lite/examples/image_classification/android_java">Android</a> and <a href="https://github.com/tensorflow/examples/tree/master/lite/examples/image_classification/ios">iOS</a> code samples demonstrate how to process full-sized camera images into the required format for each model.
+
+<h3>Uses and limitations</h3>
+
+The TensorFlow Lite image classification models are useful for single-label
+classification; that is, predicting which single label the image is most likely to
+represent. They are trained to recognize 1000 image classes. For a full list of
+classes, see the labels file in the
+<a href="https://storage.googleapis.com/download.tensorflow.org/models/tflite/mobilenet_v1_1.0_224_quant_and_labels.zip">model
+zip</a>.
+
+If you want to train a model to recognize new classes, see
+<a href="#customize_model">Customize model</a>.
+
+For the following use cases, you should use a different type of model:
+
+<ul>
+  <li>Predicting the type and position of one or more objects within an image (see <a href="../object_detection/overview">Object detection</a>)</li>
+  <li>Predicting the composition of an image, for example subject versus background (see <a href="../segmentation/overview">Segmentation</a>)</li>
+</ul>
+
+Once you have the starter model running on your target device, you can
+experiment with different models to find the optimal balance between
+performance, accuracy, and model size.
+
+<h3>Customize model</h3>
+
+The pre-trained models provided are trained to recognize 1000 classes of images.
+For a full list of classes, see the labels file in the
+<a href="https://storage.googleapis.com/download.tensorflow.org/models/tflite/mobilenet_v1_1.0_224_quant_and_labels.zip">model
+zip</a>.
+
+You can also use transfer learning to re-train a model to
+recognize classes not in the original set. For example, you could re-train the
+model to distinguish between different species of tree, despite there being no
+trees in the original training data. To do this, you will need a set of training
+images for each of the new labels you wish to train.
+
+Learn how to perform transfer learning with the
+<a href="https://www.tensorflow.org/lite/models/modify/model_maker/image_classification">TFLite Model Maker</a>,
+or in the
+<a href="https://codelabs.developers.google.com/codelabs/recognize-flowers-with-tensorflow-on-android/index.html#0">Recognize
+flowers with TensorFlow</a> codelab.
+
+<h2>Performance benchmarks</h2>
+
+Model performance is measured in terms of the amount of time it takes for a
+model to run inference on a given piece of hardware. The lower the time, the faster
+the model.
+
+The performance you require depends on your application. Performance can be
+important for applications like real-time video, where it may be important to
+analyze each frame in the time before the next frame is drawn (e.g. inference
+must be faster than 33ms to perform real-time inference on a 30fps video
+stream).
+
+The TensorFlow Lite quantized MobileNet models' performance range from 3.7ms to
+80.3 ms.
+
+Performance benchmark numbers are generated with the
+<a href="https://www.tensorflow.org/lite/performance/benchmarks">benchmarking tool</a>.
+
+<table>
+  <thead>
+    <tr>
+      <th>Model Name</th>
+      <th>Model size </th>
+      <th>Device </th>
+      <th>NNAPI</th>
+      <th>CPU</th>
+    </tr>
+  </thead>
+  <tr>
+    <td rowspan = 3>
+      <a href="https://storage.googleapis.com/download.tensorflow.org/models/tflite/mobilenet_v1_1.0_224_quant_and_labels.zip">Mobilenet_V1_1.0_224_quant</a>
+    </td>
+    <td rowspan = 3>
+      4.3 Mb
+    </td>
+    <td>Pixel 3 (Android 10) </td>
+    <td>6ms</td>
+    <td>13ms*</td>
+  </tr>
+   <tr>
+     <td>Pixel 4 (Android 10) </td>
+    <td>3.3ms</td>
+    <td>5ms*</td>
+  </tr>
+   <tr>
+     <td>iPhone XS (iOS 12.4.1) </td>
+     <td></td>
+    <td>11ms** </td>
+  </tr>
+</table>
+
+\* 4 threads used.
+
+\*\* 2 threads used on iPhone for the best performance result.
+
+### Model accuracy
+
+Accuracy is measured in terms of how often the model correctly classifies an
+image. For example, a model with a stated accuracy of 60% can be expected to
+classify an image correctly an average of 60% of the time.
+
+The most relevant accuracy metrics are Top-1 and Top-5. Top-1 refers to how
+often the correct label appears as the label with the highest probability in the
+model’s output. Top-5 refers to how often the correct label appears in the 5
+highest probabilities in the model’s output.
+
+The TensorFlow Lite quantized MobileNet models’ Top-5 accuracy range from 64.4
+to 89.9%.
+
+### Model size
+
+The size of a model on-disk varies with its performance and accuracy. Size may
+be important for mobile development (where it might impact app download sizes)
+or when working with hardware (where available storage might be limited).
+
+The TensorFlow Lite quantized MobileNet models' sizes range from 0.5 to 3.4 MB.
+
+## Further reading and resources
+
+Use the following resources to learn more about concepts related to image
+classification:
+
+*   [Image classification using TensorFlow](https://www.tensorflow.org/tutorials/images/classification)
+*   [Image classification with CNNs](https://www.tensorflow.org/tutorials/images/cnn)
+*   [Transfer learning](https://www.tensorflow.org/tutorials/images/transfer_learning)
+*   [Data augmentation](https://www.tensorflow.org/tutorials/images/data_augmentation)

tensorflow/lite/g3doc/examples/jax_conversion/jax_to_tflite.ipynb

@@ -0,0 +1,485 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "8vD3L4qeREvg"
+      },
+      "source": [
+        "##### Copyright 2024 The AI Edge Authors."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "qLCxmWRyRMZE"
+      },
+      "outputs": [],
+      "source": [
+        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
+        "# you may not use this file except in compliance with the License.\n",
+        "# You may obtain a copy of the License at\n",
+        "#\n",
+        "# https://www.apache.org/licenses/LICENSE-2.0\n",
+        "#\n",
+        "# Unless required by applicable law or agreed to in writing, software\n",
+        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
+        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
+        "# See the License for the specific language governing permissions and\n",
+        "# limitations under the License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "8LYgHRFPRpS1"
+      },
+      "source": []
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "4k5PoHrgJQOU"
+      },
+      "source": [
+        "# Jax Model Conversion For TFLite\n",
+        "## Overview\n",
+        "Note: This API is new and we recommend using via pip install tf-nightly. Also, the API is still experimental and subject to changes.\n",
+        "\n",
+        "This CodeLab demonstrates how to build a model for MNIST recognition using Jax, and how to convert it to TensorFlow Lite. This codelab will also demonstrate how to optimize the Jax-converted TFLite model with post-training quantiztion."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "i8cfOBcjSByO"
+      },
+      "source": [
+        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
+        "  <td>\n",
+        "    <a target=\"_blank\" href=\"https://www.tensorflow.org/lite/examples/jax_conversion/jax_to_tflite\"><img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" />View on TensorFlow.org</a>\n",
+        "  </td>\n",
+        "  <td>\n",
+        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/examples/jax_conversion/jax_to_tflite.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
+        "  </td>\n",
+        "  <td>\n",
+        "    <a target=\"_blank\" href=\"https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/examples/jax_conversion/jax_to_tflite.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
+        "  </td>\n",
+        "  <td>\n",
+        "    <a href=\"https://storage.googleapis.com/tensorflow_docs/tensorflow/tensorflow/lite/g3doc/examples/jax_conversion/jax_to_tflite.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\" />Download notebook</a>\n",
+        "  </td>\n",
+        "</table>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "lq-T8XZMJ-zv"
+      },
+      "source": [
+        "## Prerequisites\n",
+        "It's recommended to try this feature with the newest TensorFlow nightly pip build."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "EV04hKdrnE4f"
+      },
+      "outputs": [],
+      "source": [
+        "!pip install tf-nightly --upgrade\n",
+        "!pip install jax --upgrade"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "vsilblGuGQa2"
+      },
+      "outputs": [],
+      "source": [
+        "# Make sure your JAX version is at least 0.4.20 or above.\n",
+        "import jax\n",
+        "jax.__version__"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "PJeQhMUwH0oX"
+      },
+      "outputs": [],
+      "source": [
+        "!pip install orbax-export --upgrade"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "j9_CVA0THQNc"
+      },
+      "outputs": [],
+      "source": [
+        "from orbax.export import ExportManager\n",
+        "from orbax.export import JaxModule\n",
+        "from orbax.export import ServingConfig"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "QAeY43k9KM55"
+      },
+      "source": [
+        "## Data Preparation\n",
+        "Download the MNIST data with Keras dataset and pre-process."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "qSOPSZJn1_Tj"
+      },
+      "outputs": [],
+      "source": [
+        "import numpy as np\n",
+        "import tensorflow as tf\n",
+        "import functools\n",
+        "\n",
+        "import time\n",
+        "import itertools\n",
+        "\n",
+        "import numpy.random as npr\n",
+        "\n",
+        "import jax.numpy as jnp\n",
+        "from jax import jit, grad, random\n",
+        "from jax.example_libraries import optimizers\n",
+        "from jax.example_libraries import stax\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "hdJIt3Da2Qn1"
+      },
+      "outputs": [],
+      "source": [
+        "def _one_hot(x, k, dtype=np.float32):\n",
+        "  \"\"\"Create a one-hot encoding of x of size k.\"\"\"\n",
+        "  return np.array(x[:, None] == np.arange(k), dtype)\n",
+        "\n",
+        "(train_images, train_labels), (test_images, test_labels) = tf.keras.datasets.mnist.load_data()\n",
+        "train_images, test_images = train_images / 255.0, test_images / 255.0\n",
+        "train_images = train_images.astype(np.float32)\n",
+        "test_images = test_images.astype(np.float32)\n",
+        "\n",
+        "train_labels = _one_hot(train_labels, 10)\n",
+        "test_labels = _one_hot(test_labels, 10)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "0eFhx85YKlEY"
+      },
+      "source": [
+        "## Build the MNIST model with Jax"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "mi3TKB9nnQdK"
+      },
+      "outputs": [],
+      "source": [
+        "def loss(params, batch):\n",
+        "  inputs, targets = batch\n",
+        "  preds = predict(params, inputs)\n",
+        "  return -jnp.mean(jnp.sum(preds * targets, axis=1))\n",
+        "\n",
+        "def accuracy(params, batch):\n",
+        "  inputs, targets = batch\n",
+        "  target_class = jnp.argmax(targets, axis=1)\n",
+        "  predicted_class = jnp.argmax(predict(params, inputs), axis=1)\n",
+        "  return jnp.mean(predicted_class == target_class)\n",
+        "\n",
+        "init_random_params, predict = stax.serial(\n",
+        "    stax.Flatten,\n",
+        "    stax.Dense(1024), stax.Relu,\n",
+        "    stax.Dense(1024), stax.Relu,\n",
+        "    stax.Dense(10), stax.LogSoftmax)\n",
+        "\n",
+        "rng = random.PRNGKey(0)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "bRtnOBdJLd63"
+      },
+      "source": [
+        "## Train & Evaluate the model"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "SWbYRyj7LYZt"
+      },
+      "outputs": [],
+      "source": [
+        "step_size = 0.001\n",
+        "num_epochs = 10\n",
+        "batch_size = 128\n",
+        "momentum_mass = 0.9\n",
+        "\n",
+        "\n",
+        "num_train = train_images.shape[0]\n",
+        "num_complete_batches, leftover = divmod(num_train, batch_size)\n",
+        "num_batches = num_complete_batches + bool(leftover)\n",
+        "\n",
+        "def data_stream():\n",
+        "  rng = npr.RandomState(0)\n",
+        "  while True:\n",
+        "    perm = rng.permutation(num_train)\n",
+        "    for i in range(num_batches):\n",
+        "      batch_idx = perm[i * batch_size:(i + 1) * batch_size]\n",
+        "      yield train_images[batch_idx], train_labels[batch_idx]\n",
+        "batches = data_stream()\n",
+        "\n",
+        "opt_init, opt_update, get_params = optimizers.momentum(step_size, mass=momentum_mass)\n",
+        "\n",
+        "@jit\n",
+        "def update(i, opt_state, batch):\n",
+        "  params = get_params(opt_state)\n",
+        "  return opt_update(i, grad(loss)(params, batch), opt_state)\n",
+        "\n",
+        "_, init_params = init_random_params(rng, (-1, 28 * 28))\n",
+        "opt_state = opt_init(init_params)\n",
+        "itercount = itertools.count()\n",
+        "\n",
+        "print(\"\\nStarting training...\")\n",
+        "for epoch in range(num_epochs):\n",
+        "  start_time = time.time()\n",
+        "  for _ in range(num_batches):\n",
+        "    opt_state = update(next(itercount), opt_state, next(batches))\n",
+        "  epoch_time = time.time() - start_time\n",
+        "\n",
+        "  params = get_params(opt_state)\n",
+        "  train_acc = accuracy(params, (train_images, train_labels))\n",
+        "  test_acc = accuracy(params, (test_images, test_labels))\n",
+        "  print(\"Epoch {} in {:0.2f} sec\".format(epoch, epoch_time))\n",
+        "  print(\"Training set accuracy {}\".format(train_acc))\n",
+        "  print(\"Test set accuracy {}\".format(test_acc))"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "7Y1OZBhfQhOj"
+      },
+      "source": [
+        "## Convert to TFLite model.\n",
+        "Note here, we\n",
+        "1. Export the `JAX` model to `TF SavedModel` using `orbax`.\n",
+        "2. Call TFLite converter API to convert the `TF SavedModel` to `.tflite` model:\n",
+        "\n",
+        "\n",
+        "\n",
+        "\n",
+        "\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "6pcqKZqdNTmn"
+      },
+      "outputs": [],
+      "source": [
+        "jax_module = JaxModule(params, predict, input_polymorphic_shape='b, ...')\n",
+        "converter = tf.lite.TFLiteConverter.from_concrete_functions(\n",
+        "    [\n",
+        "        jax_module.methods[JaxModule.DEFAULT_METHOD_KEY].get_concrete_function(\n",
+        "            tf.TensorSpec(shape=(1, 28, 28), dtype=tf.float32, name=\"input\")\n",
+        "        )\n",
+        "    ]\n",
+        ")\n",
+        "\n",
+        "tflite_model = converter.convert()\n",
+        "with open('jax_mnist.tflite', 'wb') as f:\n",
+        "  f.write(tflite_model)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "sqEhzaJPSPS1"
+      },
+      "source": [
+        "## Check the Converted TFLite Model\n",
+        "Compare the converted model's results with the Jax model."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "acj2AYzjSlaY"
+      },
+      "outputs": [],
+      "source": [
+        "serving_func = functools.partial(predict, params)\n",
+        "expected = serving_func(train_images[0:1])\n",
+        "\n",
+        "# Run the model with TensorFlow Lite\n",
+        "interpreter = tf.lite.Interpreter(model_content=tflite_model)\n",
+        "interpreter.allocate_tensors()\n",
+        "input_details = interpreter.get_input_details()\n",
+        "output_details = interpreter.get_output_details()\n",
+        "interpreter.set_tensor(input_details[0][\"index\"], train_images[0:1, :, :])\n",
+        "interpreter.invoke()\n",
+        "result = interpreter.get_tensor(output_details[0][\"index\"])\n",
+        "\n",
+        "# Assert if the result of TFLite model is consistent with the JAX model.\n",
+        "np.testing.assert_almost_equal(expected, result, 1e-5)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "Qy9Gp4H2SjBL"
+      },
+      "source": [
+        "## Optimize the Model\n",
+        "We will provide a `representative_dataset` to do post-training quantiztion to optimize the model.\n",
+        "\n",
+        "\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "KI0rLV-Meg-2"
+      },
+      "outputs": [],
+      "source": [
+        "def representative_dataset():\n",
+        "  for i in range(1000):\n",
+        "    x = train_images[i:i+1]\n",
+        "    yield [x]\n",
+        "x_input = jnp.zeros((1, 28, 28))\n",
+        "converter = tf.lite.TFLiteConverter.experimental_from_jax(\n",
+        "    [serving_func], [[('x', x_input)]])\n",
+        "tflite_model = converter.convert()\n",
+        "converter.optimizations = [tf.lite.Optimize.DEFAULT]\n",
+        "converter.representative_dataset = representative_dataset\n",
+        "converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]\n",
+        "tflite_quant_model = converter.convert()\n",
+        "with open('jax_mnist_quant.tflite', 'wb') as f:\n",
+        "  f.write(tflite_quant_model)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "15xQR3JZS8TV"
+      },
+      "source": [
+        "## Evaluate the Optimized Model"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "X3oOm0OaevD6"
+      },
+      "outputs": [],
+      "source": [
+        "expected = serving_func(train_images[0:1])\n",
+        "\n",
+        "# Run the model with TensorFlow Lite\n",
+        "interpreter = tf.lite.Interpreter(model_content=tflite_quant_model)\n",
+        "interpreter.allocate_tensors()\n",
+        "input_details = interpreter.get_input_details()\n",
+        "output_details = interpreter.get_output_details()\n",
+        "interpreter.set_tensor(input_details[0][\"index\"], train_images[0:1, :, :])\n",
+        "interpreter.invoke()\n",
+        "result = interpreter.get_tensor(output_details[0][\"index\"])\n",
+        "\n",
+        "# Assert if the result of TFLite model is consistent with the Jax model.\n",
+        "np.testing.assert_almost_equal(expected, result, 1e-5)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "QqHXCNa3myor"
+      },
+      "source": [
+        "## Compare the Quantized Model size\n",
+        "We should be able to see the quantized model is four times smaller than the original model."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "imFPw007juVG"
+      },
+      "outputs": [],
+      "source": [
+        "!du -h jax_mnist.tflite\n",
+        "!du -h jax_mnist_quant.tflite"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "N5WdO18wNfyn"
+      },
+      "outputs": [],
+      "source": []
+    }
+  ],
+  "metadata": {
+    "colab": {
+      "private_outputs": true,
+      "provenance": [
+        {
+          "file_id": "1UlzbQspn2an2kzlLWZBWhP_JEqvShxmi",
+          "timestamp": 1705015454450
+        },
+        {
+          "file_id": "https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/examples/jax_conversion/overview.ipynb",
+          "timestamp": 1698963811786
+        }
+      ],
+      "toc_visible": true
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}

tensorflow/lite/g3doc/examples/jax_conversion/jax_to_tflite_resnet50.ipynb

@@ -0,0 +1,517 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "eB-dvsMI09O4"
+      },
+      "source": [
+        "##### Copyright 2024 The AI Edge Authors."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "cellView": "form",
+        "id": "mwvC53CC1K3n"
+      },
+      "outputs": [],
+      "source": [
+        "# @title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
+        "# you may not use this file except in compliance with the License.\n",
+        "# You may obtain a copy of the License at\n",
+        "#\n",
+        "# https://www.apache.org/licenses/LICENSE-2.0\n",
+        "#\n",
+        "# Unless required by applicable law or agreed to in writing, software\n",
+        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
+        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
+        "# See the License for the specific language governing permissions and\n",
+        "# limitations under the License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "0WlR2-Y3v1mf"
+      },
+      "source": [
+        "End-to-end Example to show how a pre-trained FLAX model can be downloaded, exported and ran.\n",
+        "===============\n",
+        "1. The model used is Resnet50, downloaded from huggingface pre-trained models\n",
+        "2. Test image is of a CAT\n",
+        "3. `orbax-export` API is used to export the JAX Module to a TF Saved Model, along with image pre/post-processing functions.\n",
+        "4. TFLite Converter & Runtime are used to convert the TF Saved Model to .tflite and run on the test image.\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "zQXWQ7y11eIR"
+      },
+      "source": [
+        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
+        "  <td>\n",
+        "    <a target=\"_blank\" href=\"https://www.tensorflow.org/lite/examples/jax_conversion/jax_to_tflite_resnet50\"><img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" />View on TensorFlow.org</a>\n",
+        "  </td>\n",
+        "  <td>\n",
+        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/examples/jax_conversion/jax_to_tflite_resnet50.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
+        "  </td>\n",
+        "  <td>\n",
+        "    <a target=\"_blank\" href=\"https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/examples/jax_conversion/jax_to_tflite_resnet50.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
+        "  </td>\n",
+        "  <td>\n",
+        "    <a href=\"https://storage.googleapis.com/tensorflow_docs/tensorflow/tensorflow/lite/g3doc/examples/jax_conversion/jax_to_tflite_resnet50.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\" />Download notebook</a>\n",
+        "  </td>\n",
+        "</table>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "gq1373VwVlKV"
+      },
+      "source": [
+        "# Setup"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "TUOym6hJnTxC"
+      },
+      "source": [
+        "```\n",
+        "!pip install orbax-export\n",
+        "\n",
+        "!pip install tf-nightly\n",
+        "\n",
+        "!pip install --upgrade jax jaxlib\n",
+        "\n",
+        "!pip install transformers flax\n",
+        "```"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "hDzBKqEfrafW"
+      },
+      "source": [
+        "## Show Test Image"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "2IOV4p_jripo"
+      },
+      "outputs": [],
+      "source": [
+        "from PIL import Image\n",
+        "import jax\n",
+        "import requests\n",
+        "\n",
+        "url = \"https://storage.googleapis.com/download.tensorflow.org/example_images/astrid_l_shaped.jpg\"\n",
+        "image = Image.open(requests.get(url, stream=True).raw)\n",
+        "image"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "SPcMyI8ERmIB"
+      },
+      "source": [
+        "## Download and test pre-trained Resnet50 FLAX model from HuggingFace"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "a__LrY62PUtg"
+      },
+      "outputs": [],
+      "source": [
+        "from transformers import ConvNextImageProcessor, FlaxResNetForImageClassification\n",
+        "\n",
+        "image_processor = ConvNextImageProcessor.from_pretrained(\"microsoft/resnet-50\")\n",
+        "model = FlaxResNetForImageClassification.from_pretrained(\"microsoft/resnet-50\")\n",
+        "\n",
+        "inputs = image_processor(images=image, return_tensors=\"np\")\n",
+        "outputs = model(**inputs)\n",
+        "logits = outputs.logits\n",
+        "\n",
+        "# model predicts one of the 1000 ImageNet classes\n",
+        "predicted_class_idx = jax.numpy.argmax(logits, axis=-1)\n",
+        "print(\"Predicted class:\", model.config.id2label[predicted_class_idx.item()])"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "MKJHRRjGR1rC"
+      },
+      "source": [
+        "# Create a JAX wrapper for the model\n",
+        "Wrapper is needed in order to comply with TFLite accepts inputs. TFLite accets a tensor or a tuple-of-tensors."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "Af2zv1qFSOR3"
+      },
+      "outputs": [],
+      "source": [
+        "import flax.linen as nn\n",
+        "from transformers import FlaxResNetForImageClassification\n",
+        "\n",
+        "\n",
+        "class Resnet50Wrapper(nn.Module):\n",
+        "  pretrained_model_name: str = \"microsoft/resnet-50\"  # Pre-trained model name\n",
+        "\n",
+        "  def setup(self):\n",
+        "    # Initialize the pre-trained ResNet50 model\n",
+        "    self.model = FlaxResNetForImageClassification.from_pretrained(\n",
+        "        self.pretrained_model_name\n",
+        "    )\n",
+        "\n",
+        "  def __call__(self, inputs):\n",
+        "    # Process input images through the ResNet50 model\n",
+        "    outputs = self.model(pixel_values=inputs)\n",
+        "\n",
+        "    # Return logits or directly apply softmax for probabilities (optional)\n",
+        "    return outputs.logits"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "mrKtRfkhf73M"
+      },
+      "source": [
+        "## Tensorflow image pre-processor function\n",
+        "This essentialliy implements the underlying logic of the `ConvNextImageProcessor` class in huggingface transformers:\n",
+        "> ```\n",
+        "> image_processor = ConvNextImageProcessor.from_pretrained(\"microsoft/resnet-50\")\n",
+        "> inputs = image_processor(images=image, return_tensors=\"np\")\n",
+        "> ```\n",
+        "\n",
+        "This utility can be reused later during orbax-export for `tf_preprocessing`.\n",
+        "\n",
+        "Note: We can perfectly use the result of `ConvNextImageProcessor` to run a TFLite model. But this example would like to showcase how `orbax-export` helps handle input/output pre/post-processing."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "h85-Qb1PgANA"
+      },
+      "outputs": [],
+      "source": [
+        "import tensorflow as tf\n",
+        "import numpy as np\n",
+        "\n",
+        "\n",
+        "def resnet_image_processor(image_tensor):\n",
+        "  # 1. Resize and Cast to Float32\n",
+        "  image_resized = tf.image.resize(\n",
+        "      image_tensor, (224, 224), method=tf.image.ResizeMethod.BILINEAR\n",
+        "  )\n",
+        "  image_float = tf.cast(image_resized, tf.float32)\n",
+        "\n",
+        "  # 2. Normalize (Using TensorFlow Constants)\n",
+        "  mean = tf.constant([0.485, 0.456, 0.406])\n",
+        "  std = tf.constant([0.229, 0.224, 0.225])\n",
+        "  image_normalized = (image_float / 255.0 - mean) / std\n",
+        "\n",
+        "  # 3. Transpose for Channel-First Format\n",
+        "  image_transposed = tf.transpose(image_normalized, perm=[2, 0, 1])\n",
+        "\n",
+        "  # 4. Add Batch Dimension\n",
+        "  return tf.expand_dims(image_transposed, axis=0)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "3GD4Wb2K3VT_"
+      },
+      "source": [
+        "## Initialize the Jax model"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "2wHvNFpiThVL"
+      },
+      "outputs": [],
+      "source": [
+        "# Initialize the JAX Model\n",
+        "jax_model = Resnet50Wrapper()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "uwP7XjS43M7W"
+      },
+      "source": [
+        "## Validate the wrapped JAX Model"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "Gl8mypra3MGw"
+      },
+      "outputs": [],
+      "source": [
+        "# Convert the raw image values to RGB tensor\n",
+        "raw_image_tensor = tf.convert_to_tensor(np.array(image, dtype=np.float32))\n",
+        "\n",
+        "# Appy the above TF imape preprocessing to get an input tensor supported by Resnet50\n",
+        "input_tensor = resnet_image_processor(raw_image_tensor)\n",
+        "\n",
+        "# Run the JAX model\n",
+        "jax_logits = jax_model.apply({}, input_tensor.numpy())\n",
+        "\n",
+        "jax_predicted_class_idx = jax.numpy.argmax(jax_logits, axis=-1)\n",
+        "print(\"Predicted class:\", model.config.id2label[jax_predicted_class_idx.item()])"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "lKZst83Mu6TD"
+      },
+      "outputs": [],
+      "source": [
+        "raw_image_tensor.shape"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "zuAl8ubTTKxZ"
+      },
+      "source": [
+        "# Export to TFLite model and run"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "Pz61IKnwjhbn"
+      },
+      "source": [
+        "## Export the JAX to TF Saved Model using orbax-export"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "Cwq1rxEZjpoU"
+      },
+      "outputs": [],
+      "source": [
+        "from orbax.export import ExportManager, JaxModule, ServingConfig\n",
+        "\n",
+        "# Wrap the model params and function into a JaxModule.\n",
+        "jax_module = JaxModule({}, jax_model.apply, trainable=False)\n",
+        "\n",
+        "# Specify the serving configuration and export the model.\n",
+        "serving_config = ServingConfig(\n",
+        "    \"serving_default\",\n",
+        "    input_signature=[tf.TensorSpec([480, 640, 3], tf.float32, name=\"inputs\")],\n",
+        "    tf_preprocessor=resnet_image_processor,\n",
+        "    tf_postprocessor=lambda x: tf.argmax(x, axis=-1),\n",
+        ")\n",
+        "\n",
+        "export_manager = ExportManager(jax_module, [serving_config])\n",
+        "\n",
+        "saved_model_dir = \"resnet50_saved_model\"\n",
+        "export_manager.save(saved_model_dir)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "O00xe8MevVLj"
+      },
+      "source": [
+        "### Convert to a TFLite Model"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "Uf2-ZMECmtJ3"
+      },
+      "outputs": [],
+      "source": [
+        "converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)\n",
+        "tflite_model = converter.convert()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "n-fBAd64vZse"
+      },
+      "source": [
+        "### Apply TFLite Runtime API on the `raw_image_tensor`"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "unFFby0Zmup-"
+      },
+      "outputs": [],
+      "source": [
+        "def run_tflite_model(tflite_model_content, input_tensor):\n",
+        "  interpreter = tf.lite.Interpreter(model_content=tflite_model_content)\n",
+        "  interpreter.allocate_tensors()\n",
+        "\n",
+        "  input_details = interpreter.get_input_details()[0]\n",
+        "  interpreter.set_tensor(input_details[\"index\"], input_tensor)\n",
+        "  interpreter.invoke()\n",
+        "\n",
+        "  output_details = interpreter.get_output_details()\n",
+        "  return interpreter.get_tensor(output_details[0][\"index\"])"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "Z9Rwk8iFvSBO"
+      },
+      "outputs": [],
+      "source": [
+        "output_data = run_tflite_model(tflite_model, raw_image_tensor)\n",
+        "print(\"Predicted class:\", model.config.id2label[output_data[0]])"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "NbtCrNLYKpK8"
+      },
+      "source": [
+        "## Export to TF Saved Model without pre/post-processing"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "qPLmCDniKvV3"
+      },
+      "outputs": [],
+      "source": [
+        "saved_model_dir_2 = \"resnet50_saved_model_1\"\n",
+        "\n",
+        "tf.saved_model.save(\n",
+        "    jax_module,\n",
+        "    saved_model_dir_2,\n",
+        "    signatures=jax_module.methods[JaxModule.DEFAULT_METHOD_KEY].get_concrete_function(\n",
+        "        tf.TensorSpec([1, 3, 224, 224], tf.float32, name=\"inputs\")\n",
+        "    ),\n",
+        "    options=tf.saved_model.SaveOptions(experimental_custom_gradients=True),\n",
+        ")"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "Wzp8M2ebLYoH"
+      },
+      "outputs": [],
+      "source": [
+        "converter_1 = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir_2)\n",
+        "tflite_model_1 = converter_1.convert()"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "3Di5ikJYMcKy"
+      },
+      "outputs": [],
+      "source": [
+        "output_data_1 = run_tflite_model(tflite_model_1, input_tensor)\n",
+        "tfl_predicted_class_idx_1 = tf.argmax(output_data_1, axis=-1).numpy()\n",
+        "print(\"Predicted class:\", model.config.id2label[tfl_predicted_class_idx_1[0]])"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "_SwsbUH_Smt7"
+      },
+      "source": [
+        "## Export from TF Function"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "2MbcN1vjSt5E"
+      },
+      "outputs": [],
+      "source": [
+        "converter_2 = tf.lite.TFLiteConverter.from_concrete_functions(\n",
+        "    [\n",
+        "        jax_module.methods[JaxModule.DEFAULT_METHOD_KEY].get_concrete_function(\n",
+        "            tf.TensorSpec([1, 3, 224, 224], tf.float32, name=\"inputs\")\n",
+        "        )\n",
+        "    ]\n",
+        ")\n",
+        "tflite_model_2 = converter_2.convert()"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "WdM3p03AS-xq"
+      },
+      "outputs": [],
+      "source": [
+        "output_data_2 = run_tflite_model(tflite_model_2, input_tensor)\n",
+        "tfl_predicted_class_idx_2 = tf.argmax(output_data_2, axis=-1).numpy()\n",
+        "print(\"Predicted class:\", model.config.id2label[tfl_predicted_class_idx_2[0]])"
+      ]
+    }
+  ],
+  "metadata": {
+    "colab": {
+      "name": "jax_to_tflite_resnet50.ipynb",
+      "toc_visible": true
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}

tensorflow/lite/g3doc/examples/jax_conversion/overview.md

@@ -0,0 +1,98 @@
+# JAX models with TensorFlow Lite
+
+This page provides a path for users who want to train models in JAX and deploy
+to mobile for inference ([example colab](https://colab.sandbox.google.com/github/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/examples/jax_conversion/jax_to_tflite.ipynb)).
+
+The methods in this guide produce a `tflite_model` which can be used directly
+with the TFLite interpreter code example or saved to a TFLite FlatBuffer file.
+
+## Prerequisite
+
+It's recommended to try this feature with the newest TensorFlow nightly Python
+package.
+
+```
+pip install tf-nightly --upgrade
+```
+
+We will use the
+[Orbax Export](https://orbax.readthedocs.io/en/latest/guides/export/orbax_export_101.html)
+library to export JAX models. Make sure your JAX version is at least 0.4.20 or
+above.
+
+```
+pip install jax --upgrade
+pip install orbax-export --upgrade
+```
+
+## Convert JAX models to TensorFlow Lite
+
+We use the TensorFlow
+[SavedModel](https://www.tensorflow.org/guide/saved_model) as the intermediate
+format between JAX and TensorFlow Lite. Once you have a SavedModel then
+existing TensorFlow Lite APIs can be used to complete the conversion process.
+
+```py
+# This code snippet converts a JAX model to TFLite through TF SavedModel.
+from orbax.export import ExportManager
+from orbax.export import JaxModule
+from orbax.export import ServingConfig
+import tensorflow as tf
+import jax.numpy as jnp
+
+def model_fn(_, x):
+  return jnp.sin(jnp.cos(x))
+
+jax_module = JaxModule({}, model_fn, input_polymorphic_shape='b, ...')
+
+# Option 1: Simply save the model via `tf.saved_model.save` if no need for pre/post
+# processing.
+tf.saved_model.save(
+    jax_module,
+    '/some/directory',
+    signatures=jax_module.methods[JaxModule.DEFAULT_METHOD_KEY].get_concrete_function(
+        tf.TensorSpec(shape=(None,), dtype=tf.float32, name="input")
+    ),
+    options=tf.saved_model.SaveOptions(experimental_custom_gradients=True),
+)
+converter = tf.lite.TFLiteConverter.from_saved_model('/some/directory')
+tflite_model = converter.convert()
+
+# Option 2: Define pre/post processing TF functions (e.g. (de)?tokenize).
+serving_config = ServingConfig(
+    'Serving_default',
+    # Corresponds to the input signature of `tf_preprocessor`
+    input_signature=[tf.TensorSpec(shape=(None,), dtype=tf.float32, name='input')],
+    tf_preprocessor=lambda x: x,
+    tf_postprocessor=lambda out: {'output': out}
+)
+export_mgr = ExportManager(jax_module, [serving_config])
+export_mgr.save('/some/directory')
+converter = tf.lite.TFLiteConverter.from_saved_model('/some/directory')
+tflite_model = converter.convert()
+
+# Option 3: Convert from TF concrete function directly
+converter = tf.lite.TFLiteConverter.from_concrete_functions(
+    [
+        jax_module.methods[JaxModule.DEFAULT_METHOD_KEY].get_concrete_function(
+            tf.TensorSpec(shape=(None,), dtype=tf.float32, name="input")
+        )
+    ]
+)
+tflite_model = converter.convert()
+```
+
+## Check the converted TFLite model
+
+After the model is converted to TFLite, you can run TFLite interpreter APIs to
+check model outputs.
+
+```py
+# Run the model with TensorFlow Lite
+interpreter = tf.lite.Interpreter(model_content=tflite_model)
+interpreter.allocate_tensors() input_details = interpreter.get_input_details()
+output_details = interpreter.get_output_details()
+interpreter.set_tensor(input_details[0]["index"], input_data)
+interpreter.invoke()
+result = interpreter.get_tensor(output_details[0]["index"])
+```

tensorflow/lite/g3doc/examples/keras/keras_jax_backend_to_tfl.ipynb

@@ -0,0 +1,263 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "eB-dvsMI09O4"
+      },
+      "source": [
+        "##### Copyright 2024 The TensorFlow Authors."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "cellView": "form",
+        "id": "mwvC53CC1K3n"
+      },
+      "outputs": [],
+      "source": [
+        "# @title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
+        "# you may not use this file except in compliance with the License.\n",
+        "# You may obtain a copy of the License at\n",
+        "#\n",
+        "# https://www.apache.org/licenses/LICENSE-2.0\n",
+        "#\n",
+        "# Unless required by applicable law or agreed to in writing, software\n",
+        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
+        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
+        "# See the License for the specific language governing permissions and\n",
+        "# limitations under the License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "w61LHyw7v_yx"
+      },
+      "source": [
+        "Converting Keras to TFLite (via the JAX backend)\n",
+        "=========="
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "zQXWQ7y11eIR"
+      },
+      "source": [
+        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
+        "  <td>\n",
+        "    <a target=\"_blank\" href=\"https://www.tensorflow.org/lite/examples/keras/keras_jax_backend_to_tfl\"><img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" />View on TensorFlow.org</a>\n",
+        "  </td>\n",
+        "  <td>\n",
+        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/examples/keras/keras_jax_backend_to_tfl.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
+        "  </td>\n",
+        "  <td>\n",
+        "    <a target=\"_blank\" href=\"https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/examples/keras/keras_jax_backend_to_tfl.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
+        "  </td>\n",
+        "  <td>\n",
+        "    <a href=\"https://storage.googleapis.com/tensorflow_docs/tensorflow/tensorflow/lite/g3doc/examples/keras/keras_jax_backend_to_tfl.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\" />Download notebook</a>\n",
+        "  </td>\n",
+        "</table>"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "2-t7bCE0lGsH"
+      },
+      "outputs": [],
+      "source": [
+        "import os\n",
+        "\n",
+        "os.environ[\"KERAS_BACKEND\"] = \"jax\""
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "SNUGBsILwSSs"
+      },
+      "source": [
+        "## Setup"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "Ht0UjgDxliW9"
+      },
+      "outputs": [],
+      "source": [
+        "import keras\n",
+        "import tensorflow as tf\n",
+        "import numpy as np"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "kZhJOer0wXWP"
+      },
+      "source": [
+        "## Get the test image data"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "FirUqiycez0X"
+      },
+      "outputs": [],
+      "source": [
+        "from PIL import Image\n",
+        "import requests\n",
+        "\n",
+        "url = \"https://storage.googleapis.com/download.tensorflow.org/example_images/astrid_l_shaped.jpg\"\n",
+        "image = Image.open(requests.get(url, stream=True).raw)\n",
+        "image = image.resize((224, 224))\n",
+        "input_image = np.array(image)\n",
+        "input_image = np.expand_dims(input_image, axis=0)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "CHMbP6ZWwcV_"
+      },
+      "source": [
+        "## Instatiate a Resnet50 model from the Keras models library"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "CbJyUj1IoqF6"
+      },
+      "outputs": [],
+      "source": [
+        "jax_model = keras.applications.resnet.ResNet50(include_top=True, weights=\"imagenet\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "B5yqkEEXwo13"
+      },
+      "source": [
+        "## Run the keras JAX model with the test input"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "IBHBetUqfhDA"
+      },
+      "outputs": [],
+      "source": [
+        "input_data = keras.applications.resnet50.preprocess_input(input_image)\n",
+        "jax_model_output = jax_model(input_data)\n",
+        "\n",
+        "decoded_preds = keras.applications.resnet.decode_predictions(jax_model_output, top=1)[\n",
+        "    0\n",
+        "][0]\n",
+        "print(\"Predicted class:\", decoded_preds[1])"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "HaUJGHGpw0KD"
+      },
+      "source": [
+        "## Save the Keras JAX model"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "IZ4YqZLTrGc6"
+      },
+      "outputs": [],
+      "source": [
+        "saved_model_dir = \"resnet50_saved_model\"\n",
+        "jax_model.export(saved_model_dir)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "bNSGGXfhw4uO"
+      },
+      "source": [
+        "## Convert to a TFLite model file"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "MdJz2eKqsEhA"
+      },
+      "outputs": [],
+      "source": [
+        "converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)\n",
+        "tflite_model = converter.convert()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "NZ4DjfGSWS7O"
+      },
+      "source": [
+        "## Run using TFLite Runtime"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "CtMSYAkwWWVm"
+      },
+      "outputs": [],
+      "source": [
+        "interpreter = tf.lite.Interpreter(model_content=tflite_model)\n",
+        "interpreter.allocate_tensors()\n",
+        "\n",
+        "input_details = interpreter.get_input_details()[0]\n",
+        "interpreter.set_tensor(input_details[\"index\"], input_data)\n",
+        "interpreter.invoke()\n",
+        "\n",
+        "output_details = interpreter.get_output_details()\n",
+        "output_data = interpreter.get_tensor(output_details[0][\"index\"])\n",
+        "\n",
+        "tfl_predicted_class_idx = keras.applications.resnet.decode_predictions(\n",
+        "    output_data, top=1\n",
+        ")[0][0]\n",
+        "print(\"Predicted class:\", tfl_predicted_class_idx[1])"
+      ]
+    }
+  ],
+  "metadata": {
+    "colab": {
+      "name": "keras_jax_backend_to_tfl.ipynb",
+      "provenance": [],
+      "toc_visible": true
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}

tensorflow/lite/g3doc/examples/object_detection/overview.md

@@ -0,0 +1,399 @@
+# Object detection
+
+Given an image or a video stream, an object detection model can identify which
+of a known set of objects might be present and provide information about their
+positions within the image.
+
+For example, this screenshot of the <a href="#get_started">example
+application</a> shows how two objects have been recognized and their positions
+annotated:
+
+<img src="images/android_apple_banana.png" alt="Screenshot of Android example" width="30%">
+
+Note: (1) To integrate an existing model, try
+[TensorFlow Lite Task Library](https://ai.google.dev/edge/litert/libraries/task_library/object_detector).
+(2) To customize a model, try
+[TensorFlow Lite Model Maker](https://www.tensorflow.org/lite/guide/model_maker).
+
+## Get started
+
+To learn how to use object detection in a mobile app, explore the
+<a href="#example_applications_and_guides">Example applications and guides</a>.
+
+If you are using a platform other than Android or iOS, or if you are already
+familiar with the
+<a href="https://www.tensorflow.org/api_docs/python/tf/lite">TensorFlow Lite
+APIs</a>, you can download our starter object detection model and the
+accompanying labels.
+
+<a class="button button-primary" href="https://tfhub.dev/tensorflow/lite-model/ssd_mobilenet_v1/1/metadata/1?lite-format=tflite">Download
+starter model with Metadata</a>
+
+For more information about Metadata and associated fields (eg: `labels.txt`) see
+<a href="../../models/convert/metadata#read_the_metadata_from_models">Read
+the metadata from models</a>
+
+If you want to train a custom detection model for your own task, see
+<a href="#model-customization">Model customization</a>.
+
+For the following use cases, you should use a different type of model:
+
+<ul>
+  <li>Predicting which single label the image most likely represents (see <a href="../image_classification/overview.md">image classification</a>)</li>
+  <li>Predicting the composition of an image, for example subject versus background (see <a href="../segmentation/overview.md">segmentation</a>)</li>
+</ul>
+
+### Example applications and guides
+
+If you are new to TensorFlow Lite and are working with Android or iOS, we
+recommend exploring the following example applications that can help you get
+started.
+
+#### Android
+
+You can leverage the out-of-box API from
+[TensorFlow Lite Task Library](https://ai.google.dev/edge/litert/libraries/task_library/object_detector)
+to integrate object detection models in just a few lines of code. You can also
+build your own custom inference pipeline using the
+[TensorFlow Lite Interpreter Java API](../../guide/inference#load_and_run_a_model_in_java).
+
+The Android example below demonstrates the implementation for both methods
+using
+[Task library](https://github.com/tensorflow/examples/tree/master/lite/examples/object_detection/android_play_services)
+and
+[interpreter API](https://github.com/tensorflow/examples/tree/eb925e460f761f5ed643d17f0c449e040ac2ac45/lite/examples/object_detection/android/lib_interpreter),
+respectively.
+
+<a class="button button-primary" href="https://github.com/tensorflow/examples/tree/master/lite/examples/object_detection/android">View
+Android example</a>
+
+#### iOS
+
+You can integrate the model using the
+[TensorFlow Lite Interpreter Swift API](../../guide/inference#load_and_run_a_model_in_swift).
+See the iOS example below.
+
+<a class="button button-primary" href="https://github.com/tensorflow/examples/tree/master/lite/examples/object_detection/ios">View
+iOS example</a>
+
+## Model description
+
+This section describes the signature for
+[Single-Shot Detector](https://arxiv.org/abs/1512.02325) models converted to
+TensorFlow Lite from the
+[TensorFlow Object Detection API](https://github.com/tensorflow/models/blob/master/research/object_detection/).
+
+An object detection model is trained to detect the presence and location of
+multiple classes of objects. For example, a model might be trained with images
+that contain various pieces of fruit, along with a _label_ that specifies the
+class of fruit they represent (e.g. an apple, a banana, or a strawberry), and
+data specifying where each object appears in the image.
+
+When an image is subsequently provided to the model, it will output a list of
+the objects it detects, the location of a bounding box that contains each
+object, and a score that indicates the confidence that detection was correct.
+
+### Input Signature
+
+The model takes an image as input.
+
+Lets assume the expected image is 300x300 pixels, with three channels (red,
+blue, and green) per pixel. This should be fed to the model as a flattened
+buffer of 270,000 byte values (300x300x3). If the model is
+<a href="../../performance/post_training_quantization.md">quantized</a>, each
+value should be a single byte representing a value between 0 and 255.
+
+You can take a look at our
+[example app code](https://github.com/tensorflow/examples/tree/master/lite/examples/object_detection/android)
+to understand how to do this pre-processing on Android.
+
+### Output Signature
+
+The model outputs four arrays, mapped to the indices 0-4. Arrays 0, 1, and 2
+describe `N` detected objects, with one element in each array corresponding to
+each object.
+
+<table>
+  <thead>
+    <tr>
+      <th>Index</th>
+      <th>Name</th>
+      <th>Description</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td>0</td>
+      <td>Locations</td>
+      <td>Multidimensional array of [N][4] floating point values between 0 and 1, the inner arrays representing bounding boxes in the form [top, left, bottom, right]</td>
+    </tr>
+    <tr>
+      <td>1</td>
+      <td>Classes</td>
+      <td>Array of N integers (output as floating point values) each indicating the index of a class label from the labels file</td>
+    </tr>
+    <tr>
+      <td>2</td>
+      <td>Scores</td>
+      <td>Array of N floating point values between 0 and 1 representing probability that a class was detected</td>
+    </tr>
+    <tr>
+      <td>3</td>
+      <td>Number of detections</td>
+      <td>Integer value of N</td>
+    </tr>
+  </tbody>
+</table>
+
+NOTE: The number of results (10 in the above case) is a parameter set while
+exporting the detection model to TensorFlow Lite. See
+<a href="#model-customization">Model customization</a> for more details.
+
+For example, imagine a model has been trained to detect apples, bananas, and
+strawberries. When provided an image, it will output a set number of detection
+results - in this example, 5.
+
+<table style="width: 60%;">
+  <thead>
+    <tr>
+      <th>Class</th>
+      <th>Score</th>
+      <th>Location</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td>Apple</td>
+      <td>0.92</td>
+      <td>[18, 21, 57, 63]</td>
+    </tr>
+    <tr>
+      <td>Banana</td>
+      <td>0.88</td>
+      <td>[100, 30, 180, 150]</td>
+    </tr>
+    <tr>
+      <td>Strawberry</td>
+      <td>0.87</td>
+      <td>[7, 82, 89, 163] </td>
+    </tr>
+    <tr>
+      <td>Banana</td>
+      <td>0.23</td>
+      <td>[42, 66, 57, 83]</td>
+    </tr>
+    <tr>
+      <td>Apple</td>
+      <td>0.11</td>
+      <td>[6, 42, 31, 58]</td>
+    </tr>
+  </tbody>
+</table>
+
+#### Confidence score
+
+To interpret these results, we can look at the score and the location for each
+detected object. The score is a number between 0 and 1 that indicates confidence
+that the object was genuinely detected. The closer the number is to 1, the more
+confident the model is.
+
+Depending on your application, you can decide a cut-off threshold below which
+you will discard detection results. For the current example, a sensible cut-off
+is a score of 0.5 (meaning a 50% probability that the detection is valid). In
+that case, the last two objects in the array would be ignored because those
+confidence scores are below 0.5:
+
+<table style="width: 60%;">
+  <thead>
+    <tr>
+      <th>Class</th>
+      <th>Score</th>
+      <th>Location</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td>Apple</td>
+      <td>0.92</td>
+      <td>[18, 21, 57, 63]</td>
+    </tr>
+    <tr>
+      <td>Banana</td>
+      <td>0.88</td>
+      <td>[100, 30, 180, 150]</td>
+    </tr>
+    <tr>
+      <td>Strawberry</td>
+      <td>0.87</td>
+      <td>[7, 82, 89, 163] </td>
+    </tr>
+    <tr>
+      <td style="background-color: #e9cecc; text-decoration-line: line-through;">Banana</td>
+      <td style="background-color: #e9cecc; text-decoration-line: line-through;">0.23</td>
+      <td style="background-color: #e9cecc; text-decoration-line: line-through;">[42, 66, 57, 83]</td>
+    </tr>
+    <tr>
+      <td style="background-color: #e9cecc; text-decoration-line: line-through;">Apple</td>
+      <td style="background-color: #e9cecc; text-decoration-line: line-through;">0.11</td>
+      <td style="background-color: #e9cecc; text-decoration-line: line-through;">[6, 42, 31, 58]</td>
+    </tr>
+  </tbody>
+</table>
+
+The cut-off you use should be based on whether you are more comfortable with
+false positives (objects that are wrongly identified, or areas of the image that
+are erroneously identified as objects when they are not), or false negatives
+(genuine objects that are missed because their confidence was low).
+
+For example, in the following image, a pear (which is not an object that the
+model was trained to detect) was misidentified as a "person". This is an example
+of a false positive that could be ignored by selecting an appropriate cut-off.
+In this case, a cut-off of 0.6 (or 60%) would comfortably exclude the false
+positive.
+
+<img src="images/false_positive.png" alt="Screenshot of Android example showing a false positive" width="30%">
+
+#### Location
+
+For each detected object, the model will return an array of four numbers
+representing a bounding rectangle that surrounds its position. For the starter
+model provided, the numbers are ordered as follows:
+
+<table style="width: 50%; margin: 0 auto;">
+  <tbody>
+    <tr style="border-top: none;">
+      <td>[</td>
+      <td>top,</td>
+      <td>left,</td>
+      <td>bottom,</td>
+      <td>right</td>
+      <td>]</td>
+    </tr>
+  </tbody>
+</table>
+
+The top value represents the distance of the rectangle’s top edge from the top
+of the image, in pixels. The left value represents the left edge’s distance from
+the left of the input image. The other values represent the bottom and right
+edges in a similar manner.
+
+Note: Object detection models accept input images of a specific size. This is likely to be different from the size of the raw image captured by your device’s camera, and you will have to write code to crop and scale your raw image to fit the model’s input size (there are examples of this in our <a href="#get_started">example applications</a>).<br /><br />The pixel values output by the model refer to the position in the cropped and scaled image, so you must scale them to fit the raw image in order to interpret them correctly.
+
+## Performance benchmarks
+
+Performance benchmark numbers for our
+<a class="button button-primary" href="https://tfhub.dev/tensorflow/lite-model/ssd_mobilenet_v1/1/metadata/1?lite-format=tflite">starter
+model</a> are generated with the tool
+[described here](https://www.tensorflow.org/lite/performance/benchmarks).
+
+<table>
+  <thead>
+    <tr>
+      <th>Model Name</th>
+      <th>Model size </th>
+      <th>Device </th>
+      <th>GPU</th>
+      <th>CPU</th>
+    </tr>
+  </thead>
+  <tr>
+    <td rowspan = 3>
+      <a href="https://tfhub.dev/tensorflow/lite-model/ssd_mobilenet_v1/1/metadata/1?lite-format=tflite">COCO SSD MobileNet v1</a>
+    </td>
+    <td rowspan = 3>
+      27 Mb
+    </td>
+    <td>Pixel 3 (Android 10) </td>
+    <td>22ms</td>
+    <td>46ms*</td>
+  </tr>
+   <tr>
+     <td>Pixel 4 (Android 10) </td>
+    <td>20ms</td>
+    <td>29ms*</td>
+  </tr>
+   <tr>
+     <td>iPhone XS (iOS 12.4.1) </td>
+     <td>7.6ms</td>
+    <td>11ms** </td>
+  </tr>
+</table>
+
+\* 4 threads used.
+
+\*\* 2 threads used on iPhone for the best performance result.
+
+## Model Customization
+
+### Pre-trained models
+
+Mobile-optimized detection models with a variety of latency and precision
+characteristics can be found in the
+[Detection Zoo](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf1_detection_zoo.md#mobile-models).
+Each one of them follows the input and output signatures described in the
+following sections.
+
+Most of the download zips contain a `model.tflite` file. If there isn't one, a
+TensorFlow Lite flatbuffer can be generated using
+[these instructions](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/running_on_mobile_tensorflowlite.md).
+SSD models from the
+[TF2 Object Detection Zoo](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf2_detection_zoo.md)
+can also be converted to TensorFlow Lite using the instructions
+[here](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/running_on_mobile_tf2.md).
+It is important to note that detection models cannot be converted directly using
+the [TensorFlow Lite Converter](../../models/convert), since
+they require an intermediate step of generating a mobile-friendly source model.
+The scripts linked above perform this step.
+
+Both the
+[TF1](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/running_on_mobile_tensorflowlite.md)
+&
+[TF2](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/running_on_mobile_tf2.md)
+exporting scripts have parameters that can enable a larger number of output
+objects or slower, more-accurate post processing. Please use `--help` with the
+scripts to see an exhaustive list of supported arguments.
+
+> Currently, on-device inference is only optimized with SSD models. Better
+> support for other architectures like CenterNet and EfficientDet is being
+> investigated.
+
+### How to choose a model to customize?
+
+Each model comes with its own precision (quantified by mAP value) and latency
+characteristics. You should choose a model that works the best for your use-case
+and intended hardware. For example, the
+[Edge TPU](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf1_detection_zoo.md#pixel4-edge-tpu-models)
+models are ideal for inference on Google's Edge TPU on Pixel 4.
+
+You can use our
+[benchmark tool](https://www.tensorflow.org/lite/performance/measurement) to
+evaluate models and choose the most efficient option available.
+
+## Fine-tuning models on custom data
+
+The pre-trained models we provide are trained to detect 90 classes of objects.
+For a full list of classes, see the labels file in the
+<a href="https://tfhub.dev/tensorflow/lite-model/ssd_mobilenet_v1/1/metadata/1?lite-format=tflite">model
+metadata</a>.
+
+You can use a technique known as transfer learning to re-train a model to
+recognize classes not in the original set. For example, you could re-train the
+model to detect multiple types of vegetable, despite there only being one
+vegetable in the original training data. To do this, you will need a set of
+training images for each of the new labels you wish to train. The recommended
+way is to use
+[TensorFlow Lite Model Maker](https://www.tensorflow.org/lite/guide/model_maker)
+library which simplifies the process of training a TensorFlow Lite model using
+custom dataset, with a few lines of codes. It uses transfer learning to reduce
+the amount of required training data and time. You can also learn from
+[Few-shot detection Colab](https://github.com/tensorflow/models/blob/master/research/object_detection/colab_tutorials/eager_few_shot_od_training_tflite.ipynb)
+as an example of fine-tuning a pre-trained model with few examples.
+
+For fine-tuning with larger datasets, take a look at the these guides for
+training your own models with the TensorFlow Object Detection API:
+[TF1](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf1_training_and_evaluation.md),
+[TF2](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf2_training_and_evaluation.md).
+Once trained, they can be converted to a TFLite-friendly format with the
+instructions here:
+[TF1](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/running_on_mobile_tensorflowlite.md),
+[TF2](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/running_on_mobile_tf2.md)

tensorflow/lite/g3doc/examples/optical_character_recognition/overview.md

@@ -0,0 +1,127 @@
+# Optical character recognition (OCR)
+
+Optical character recognition (OCR) is the process of recognizing characters
+from images using computer vision and machine learning techniques. This
+reference app demos how to use TensorFlow Lite to do OCR. It uses a combination
+of
+[text detection model](https://tfhub.dev/sayakpaul/lite-model/east-text-detector/fp16/1)
+and a
+[text recognition model](https://tfhub.dev/tulasiram58827/lite-model/keras-ocr/float16/2)
+as an OCR pipeline to recognize text characters.
+
+## Get started
+
+<img src="images/screenshot.gif" class="attempt-right" style="max-width: 300px">
+
+If you are new to TensorFlow Lite and are working with Android, we recommend
+exploring the following example application that can help you get started.
+
+<a class="button button-primary" href="https://github.com/tensorflow/examples/tree/master/lite/examples/optical_character_recognition/android">Android
+example</a>
+
+If you are using a platform other than Android, or you are already familiar with
+the [TensorFlow Lite APIs](https://www.tensorflow.org/api_docs/python/tf/lite),
+you can download the models from [TF Hub](https://tfhub.dev/).
+
+## How it works
+
+OCR tasks are often broken down into 2 stages. First, we use a text detection
+model to detect the bounding boxes around possible texts. Second, we feed
+processed bounding boxes into a text recognition model to determine specific
+characters inside the bounding boxes (we also need to do Non-Maximal
+Suppression, perspective transformation and etc. beforing text recognition). In
+our case, both models are from TensorFlow Hub and they are FP16 quantized
+models.
+
+## Performance benchmarks
+
+Performance benchmark numbers are generated with the tool described
+[here](https://www.tensorflow.org/lite/performance/benchmarks).
+
+<table>
+  <thead>
+    <tr>
+      <th>Model Name</th>
+      <th>Model size </th>
+      <th>Device </th>
+      <th>CPU</th>
+      <th>GPU</th>
+    </tr>
+  </thead>
+  <tr>
+    <td>
+      <a href="https://tfhub.dev/sayakpaul/lite-model/east-text-detector/fp16/1">Text Detection</a>
+    </td>
+    <td>45.9 Mb</td>
+     <td>Pixel 4 (Android 10)</td>
+     <td>181.93ms*</td>
+     <td>89.77ms*</td>
+  </tr>
+  <tr>
+    <td>
+      <a href="https://tfhub.dev/tulasiram58827/lite-model/keras-ocr/float16/2">Text Recognition</a>
+    </td>
+    <td>16.8 Mb</td>
+     <td>Pixel 4 (Android 10)</td>
+     <td>338.33ms*</td>
+     <td>N/A**</td>
+  </tr>
+</table>
+
+\* 4 threads used.
+
+\** this model could not use GPU delegate since we need TensorFlow ops to run it
+
+## Inputs
+
+The text detection model accepts a 4-D `float32` Tensor of (1, 320, 320, 3) as
+input.
+
+The text recognition model accepts a 4-D `float32` Tensor of (1, 31, 200, 1) as
+input.
+
+## Outputs
+
+The text detection model returns a 4-D `float32` Tensor of shape (1, 80, 80, 5)
+as bounding box and a 4-D `float32` Tensor of shape (1,80, 80, 5) as detection
+score.
+
+The text recognition model returns a 2-D `float32` Tensor of shape (1, 48) as
+the mapping indices to the alphabet list '0123456789abcdefghijklmnopqrstuvwxyz'
+
+## Limitations
+
+*   The current
+    [text recognition model](https://tfhub.dev/tulasiram58827/lite-model/keras-ocr/float16/2)
+    is trained using synthetic data with English letters and numbers, so only
+    English is supported.
+
+*   The models are not general enough for OCR in the wild (say, random images
+    taken by a smartphone camera in a low lighting condition).
+
+So we have chosen 3 Google product logos only to demonstrate how to do OCR with
+TensorFlow Lite. If you are looking for a ready-to-use production-grade OCR
+product, you should consider
+[Google ML Kit](https://developers.google.com/ml-kit/vision/text-recognition).
+ML Kit, which uses TFLite underneath, should be sufficient for most OCR use
+cases, but there are some cases where you may want to build your own OCR
+solution with TFLite. Some examples are:
+
+*   You have your own text detection/recognition TFLite models that you would
+    like to use
+*   You have special business requirements (i.e., recognizing texts that are
+    upside down) and need to customize the OCR pipeline
+*   You want to support languages not covered by ML Kit
+*   Your target user devices don’t necessarily have Google Play services
+    installed
+
+## References
+
+*   OpenCV text detection/recognition example:
+    https://github.com/opencv/opencv/blob/master/samples/dnn/text_detection.cpp
+*   OCR TFLite community project by community contributors:
+    https://github.com/tulasiram58827/ocr_tflite
+*   OpenCV text detection:
+    https://www.pyimagesearch.com/2018/08/20/opencv-text-detection-east-text-detector/
+*   Deep Learning based Text Detection Using OpenCV:
+    https://learnopencv.com/deep-learning-based-text-detection-using-opencv-c-python/

tensorflow/lite/g3doc/examples/pose_estimation/overview.md

@@ -0,0 +1,249 @@
+# Pose estimation
+
+<img src="../images/pose.png" class="attempt-right" />
+
+Pose estimation is the task of using an ML model to estimate the pose of a
+person from an image or a video by estimating the spatial locations of key body
+joints (keypoints).
+
+## Get started
+
+If you are new to TensorFlow Lite and are working with Android or iOS, explore
+the following example applications that can help you get started.
+
+<a class="button button-primary" href="https://github.com/tensorflow/examples/tree/master/lite/examples/pose_estimation/android">
+Android example</a>
+<a class="button button-primary" href="https://github.com/tensorflow/examples/tree/master/lite/examples/pose_estimation/ios">
+iOS example</a>
+
+If you are familiar with the
+[TensorFlow Lite APIs](https://www.tensorflow.org/api_docs/python/tf/lite),
+download the starter MoveNet pose estimation model and supporting files.
+
+<a class="button button-primary" href="https://tfhub.dev/s?q=movenet"> Download
+starter model</a>
+
+If you want to try pose estimation on a web browser, check out the
+<a href="https://storage.googleapis.com/tfjs-models/demos/pose-detection/index.html?model=movenet">
+TensorFlow JS Demo</a>.
+
+## Model description
+
+### How it works
+
+Pose estimation refers to computer vision techniques that detect human figures
+in images and videos, so that one could determine, for example, where someone’s
+elbow shows up in an image. It is important to be aware of the fact that pose
+estimation merely estimates where key body joints are and does not recognize who
+is in an image or video.
+
+The pose estimation models takes a processed camera image as the input and
+outputs information about keypoints. The keypoints detected are indexed by a
+part ID, with a confidence score between 0.0 and 1.0. The confidence score
+indicates the probability that a keypoint exists in that position.
+
+We provides reference implementation of two TensorFlow Lite pose estimation
+models:
+
+*   MoveNet: the state-of-the-art pose estimation model available in two
+    flavors: Lighting and Thunder. See a comparison between these two in the
+    section below.
+*   PoseNet: the previous generation pose estimation model released in 2017.
+
+The various body joints detected by the pose estimation model are tabulated
+below:
+
+<table style="width: 30%;">
+  <thead>
+    <tr>
+      <th>Id</th>
+      <th>Part</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td>0</td>
+      <td>nose</td>
+    </tr>
+    <tr>
+      <td>1</td>
+      <td>leftEye</td>
+    </tr>
+    <tr>
+      <td>2</td>
+      <td>rightEye</td>
+    </tr>
+    <tr>
+      <td>3</td>
+      <td>leftEar</td>
+    </tr>
+    <tr>
+      <td>4</td>
+      <td>rightEar</td>
+    </tr>
+    <tr>
+      <td>5</td>
+      <td>leftShoulder</td>
+    </tr>
+    <tr>
+      <td>6</td>
+      <td>rightShoulder</td>
+    </tr>
+    <tr>
+      <td>7</td>
+      <td>leftElbow</td>
+    </tr>
+    <tr>
+      <td>8</td>
+      <td>rightElbow</td>
+    </tr>
+    <tr>
+      <td>9</td>
+      <td>leftWrist</td>
+    </tr>
+    <tr>
+      <td>10</td>
+      <td>rightWrist</td>
+    </tr>
+    <tr>
+      <td>11</td>
+      <td>leftHip</td>
+    </tr>
+    <tr>
+      <td>12</td>
+      <td>rightHip</td>
+    </tr>
+    <tr>
+      <td>13</td>
+      <td>leftKnee</td>
+    </tr>
+    <tr>
+      <td>14</td>
+      <td>rightKnee</td>
+    </tr>
+    <tr>
+      <td>15</td>
+      <td>leftAnkle</td>
+    </tr>
+    <tr>
+      <td>16</td>
+      <td>rightAnkle</td>
+    </tr>
+  </tbody>
+</table>
+
+An example output is shown below:
+
+<img alt="Animation showing pose estimation" src="https://storage.googleapis.com/download.tensorflow.org/example_images/movenet_demo.gif"/>
+
+## Performance benchmarks
+
+MoveNet is available in two flavors:
+
+*   MoveNet.Lightning is smaller, faster but less accurate than the Thunder
+    version. It can run in realtime on modern smartphones.
+*   MoveNet.Thunder is the more accurate version but also larger and slower than
+    Lightning. It is useful for the use cases that require higher accuracy.
+
+MoveNet outperforms PoseNet on a variety of datasets, especially in images with
+fitness action images. Therefore, we recommend using MoveNet over PoseNet.
+
+Performance benchmark numbers are generated with the tool
+[described here](../../performance/measurement.md). Accuracy (mAP) numbers are
+measured on a subset of the [COCO dataset](https://cocodataset.org/#home) in
+which we filter and crop each image to contain only one person .
+
+<table>
+<thead>
+  <tr>
+    <th rowspan="2">Model</th>
+    <th rowspan="2">Size (MB)</th>
+    <th rowspan="2">mAP</th>
+    <th colspan="3">Latency (ms)</th>
+  </tr>
+  <tr>
+    <td>Pixel 5 - CPU 4 threads</td>
+    <td>Pixel 5 - GPU</td>
+    <td>Raspberry Pi 4 - CPU 4 threads</td>
+  </tr>
+</thead>
+<tbody>
+  <tr>
+    <td>
+      <a href="https://tfhub.dev/google/lite-model/movenet/singlepose/thunder/tflite/float16/4">MoveNet.Thunder (FP16 quantized)</a>
+    </td>
+    <td>12.6MB</td>
+    <td>72.0</td>
+    <td>155ms</td>
+    <td>45ms</td>
+    <td>594ms</td>
+  </tr>
+  <tr>
+    <td>
+      <a href="https://tfhub.dev/google/lite-model/movenet/singlepose/thunder/tflite/int8/4">MoveNet.Thunder (INT8 quantized)</a>
+    </td>
+    <td>7.1MB</td>
+    <td>68.9</td>
+    <td>100ms</td>
+    <td>52ms</td>
+    <td>251ms</td>
+  </tr>
+  <tr>
+    <td>
+      <a href="https://tfhub.dev/google/lite-model/movenet/singlepose/lightning/tflite/float16/4">MoveNet.Lightning (FP16 quantized)</a>
+    </td>
+    <td>4.8MB</td>
+    <td>63.0</td>
+    <td>60ms</td>
+    <td>25ms</td>
+    <td>186ms</td>
+  </tr>
+  <tr>
+    <td>
+      <a href="https://tfhub.dev/google/lite-model/movenet/singlepose/lightning/tflite/int8/4">MoveNet.Lightning (INT8 quantized)</a>
+    </td>
+    <td>2.9MB</td>
+    <td>57.4</td>
+    <td>52ms</td>
+    <td>28ms</td>
+    <td>95ms</td>
+  </tr>
+  <tr>
+    <td>
+      <a href="https://storage.googleapis.com/download.tensorflow.org/models/tflite/posenet_mobilenet_v1_100_257x257_multi_kpt_stripped.tflite">PoseNet(MobileNetV1 backbone, FP32)</a>
+    </td>
+    <td>13.3MB</td>
+    <td>45.6</td>
+    <td>80ms</td>
+    <td>40ms</td>
+    <td>338ms</td>
+  </tr>
+</tbody>
+</table>
+
+## Further reading and resources
+
+*   Check out this
+    [blog post](https://blog.tensorflow.org/2021/08/pose-estimation-and-classification-on-edge-devices-with-MoveNet-and-TensorFlow-Lite.html)
+    to learn more about pose estimation using MoveNet and TensorFlow Lite.
+*   Check out this
+    [blog post](https://blog.tensorflow.org/2021/05/next-generation-pose-detection-with-movenet-and-tensorflowjs.html)
+    to learn more about pose estimation on the web.
+*   Check out this [tutorial](https://www.tensorflow.org/hub/tutorials/movenet)
+    to learn about running MoveNet on Python using a model from TensorFlow Hub.
+*   Coral/EdgeTPU can make pose estimation run much faster on edge devices. See
+    [EdgeTPU-optimized models](https://coral.ai/models/pose-estimation/) for
+    more details.
+*   Read the PoseNet paper [here](https://arxiv.org/abs/1803.08225)
+
+Also, check out these use cases of pose estimation.
+
+<ul>
+  <li><a href="https://vimeo.com/128375543">‘PomPom Mirror’</a></li>
+  <li><a href="https://youtu.be/I5__9hq-yas">Amazing Art Installation Turns You Into A Bird | Chris Milk "The Treachery of Sanctuary"</a></li>
+  <li><a href="https://vimeo.com/34824490">Puppet Parade - Interactive Kinect Puppets</a></li>
+  <li><a href="https://vimeo.com/2892576">Messa di Voce (Performance), Excerpts</a></li>
+  <li><a href="https://www.instagram.com/p/BbkKLiegrTR/">Augmented reality</a></li>
+  <li><a href="https://www.instagram.com/p/Bg1EgOihgyh/">Interactive animation</a></li>
+  <li><a href="https://www.runnersneed.com/expert-advice/gear-guides/gait-analysis.html">Gait analysis</a></li>
+</ul>

tensorflow/lite/g3doc/examples/recommendation/overview.md

@@ -0,0 +1,202 @@
+# Recommendation
+
+<table class="tfo-notebook-buttons" align="left">
+  <td>
+    <a target="_blank" href="https://www.tensorflow.org/lite/examples/recommendation/overview"><img src="https://www.tensorflow.org/images/tf_logo_32px.png" />View on TensorFlow.org</a>
+  </td>
+  {% dynamic if request.tld != 'cn' %}<td>
+    <a target="_blank" href="https://colab.research.google.com/github/tensorflow/examples/blob/master/lite/examples/recommendation/ml/ondevice_recommendation.ipynb"><img src="https://www.tensorflow.org/images/colab_logo_32px.png" />Run in Google Colab</a>
+  </td>{% dynamic endif %}
+  <td>
+    <a target="_blank" href="https://github.com/tensorflow/examples/blob/master/lite/examples/recommendation/ml/ondevice_recommendation.ipynb"><img src="https://www.tensorflow.org/images/GitHub-Mark-32px.png" />View source on GitHub</a>
+  </td>
+</table>
+
+Personalized recommendations are widely used for a variety of use cases on
+mobile devices, such as media content retrieval, shopping product suggestion,
+and next app recommendation. If you are interested in providing personalized
+recommendations in your application while respecting user privacy, we recommend
+exploring the following example and toolkit.
+
+Note: To customize a model, try
+[TensorFlow Lite Model Maker](https://www.tensorflow.org/lite/guide/model_maker).
+
+## Get started
+
+<img src="images/screenshot.gif" class="attempt-right" style="max-width: 300px">
+
+We provide a TensorFlow Lite sample application that demonstrates how to
+recommend relevant items to users on Android.
+
+<a class="button button-primary" href="https://github.com/tensorflow/examples/tree/master/lite/examples/recommendation/android">Android
+example</a>
+
+If you are using a platform other than Android, or you are already familiar with
+the TensorFlow Lite APIs, you can download our starter recommendation model.
+
+<a class="button button-primary" href="https://storage.googleapis.com/download.tensorflow.org/models/tflite/recommendation/20200720/recommendation.tar.gz">Download
+starter model</a>
+
+We also provide training script in Github to train your own model in a
+configurable way.
+
+<a class="button button-primary" href="https://github.com/tensorflow/examples/tree/master/lite/examples/recommendation/ml">Training
+code</a>
+
+## Understand the model architecture
+
+We leverage a dual-encoder model architecture, with context-encoder to encode
+sequential user history and label-encoder to encode predicted recommendation
+candidate. Similarity between context and label encodings is used to represent
+the likelihood that the predicted candidate meets the user's needs.
+
+Three different sequential user history encoding techniques are provided with
+this code base:
+
+*   Bag-of-words encoder (BOW): averaging user activities' embeddings without
+    considering context order.
+*   Convolutional neural network encoder (CNN): applying multiple layers of
+    convolutional neural networks to generate context encoding.
+*   Recurrent neural network encoder (RNN): applying recurrent neural network to
+    encode context sequence.
+
+To model each user activity, we could use the ID of the activity item (ID-based)
+, or multiple features of the item (feature-based), or a combination of both.
+The feature-based model utilizing multiple features to collectively encode
+users’ behavior. With this code base, you could create either ID-based or
+feature-based models in a configurable way.
+
+After training, a TensorFlow Lite model will be exported which can directly
+provide top-K predictions among the recommendation candidates.
+
+## Use your training data
+
+In addition to the trained model, we provide an open-sourced
+[toolkit in GitHub](https://github.com/tensorflow/examples/tree/master/lite/examples/recommendation/ml)
+to train models with your own data. You can follow this tutorial to learn how to
+use the toolkit and deploy trained models in your own mobile applications.
+
+Please follow this
+[tutorial](https://github.com/tensorflow/examples/tree/master/lite/examples/recommendation/ml/ondevice_recommendation.ipynb)
+to apply the same technique used here to train a recommendation model using your
+own datasets.
+
+## Examples
+
+As examples, we trained recommendation models with both ID-based and
+feature-based approaches. The ID-based model takes only the movie IDs as input,
+and the feature-based model takes both movie IDs and movie genre IDs as inputs.
+Please find the following inputs and outputs examples.
+
+Inputs
+
+*   Context movie IDs:
+
+    *   The Lion King (ID: 362)
+    *   Toy Story (ID: 1)
+    *   (and more)
+
+*   Context movie genre IDs:
+
+    *   Animation (ID: 15)
+    *   Children's (ID: 9)
+    *   Musical (ID: 13)
+    *   Animation (ID: 15)
+    *   Children's (ID: 9)
+    *   Comedy (ID: 2)
+    *   (and more)
+
+Outputs:
+
+*   Recommended movie IDs:
+    *   Toy Story 2 (ID: 3114)
+    *   (and more)
+
+Note: The pretrained model is built based on
+[MovieLens](https://grouplens.org/datasets/movielens/1m/) dataset for research
+purpose.
+
+## Performance benchmarks
+
+Performance benchmark numbers are generated with the tool
+[described here](https://www.tensorflow.org/lite/performance/benchmarks).
+
+<table>
+  <thead>
+    <tr>
+      <th>Model Name</th>
+      <th>Model Size </th>
+      <th>Device </th>
+      <th>CPU</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <tr>
+        <td rowspan = 3>
+          <a href="https://storage.googleapis.com/download.tensorflow.org/models/tflite/recommendation/20200720/model.tar.gz">recommendation (movie ID as input)</a>
+        </td>
+        <td rowspan = 3>
+          0.52 Mb
+        </td>
+        <td>Pixel 3</td>
+        <td>0.09ms*</td>
+      </tr>
+       <tr>
+         <td>Pixel 4 </td>
+        <td>0.05ms*</td>
+      </tr>
+    </tr>
+    <tr>
+      <tr>
+        <td rowspan = 3>
+          <a href="https://storage.googleapis.com/download.tensorflow.org/models/tflite/recommendation/20210317/recommendation_cnn_i10i32o100.tflite">recommendation (movie ID and movie genre as inputs)</a>
+        </td>
+        <td rowspan = 3>
+          1.3 Mb
+        </td>
+        <td>Pixel 3</td>
+        <td>0.13ms*</td>
+      </tr>
+       <tr>
+         <td>Pixel 4 </td>
+        <td>0.06ms*</td>
+      </tr>
+    </tr>
+  </tbody>
+</table>
+
+\* 4 threads used.
+
+## Use your training data
+
+In addition to the trained model, we provide an open-sourced
+[toolkit in GitHub](https://github.com/tensorflow/examples/tree/master/lite/examples/recommendation/ml)
+to train models with your own data. You can follow this tutorial to learn how to
+use the toolkit and deploy trained models in your own mobile applications.
+
+Please follow this
+[tutorial](https://github.com/tensorflow/examples/tree/master/lite/examples/recommendation/ml/ondevice_recommendation.ipynb)
+to apply the same technique used here to train a recommendation model using your
+own datasets.
+
+## Tips for model customization with your data
+
+The pretrained model integrated in this demo application is trained with
+[MovieLens](https://grouplens.org/datasets/movielens/1m/) dataset, you may want
+to modify model configuration based on your own data, such as vocab size,
+embedding dims and input context length. Here are a few tips:
+
+*   Input context length: The best input context length varies with datasets. We
+    suggest selecting input context length based on how much label events are
+    correlated with long-term interests vs short-term context.
+
+*   Encoder type selection: we suggest selecting encoder type based on input
+    context length. Bag-of-words encoder works well for short input context
+    length (e.g. <10), CNN and RNN encoders bring in more summarization ability
+    for long input context length.
+
+*   Using underlying features to represent items or user activities could
+    improve model performance, better accommodate fresh items, possibly down
+    scale embedding spaces hence reduce memory consumption and more on-device
+    friendly.

tensorflow/lite/g3doc/examples/reinforcement_learning/overview.md

@@ -0,0 +1,89 @@
+# Reinforcement Learning
+
+Play a board game against an agent, which is trained using reinforcement
+learning and deployed with TensorFlow Lite.
+
+## Get started
+
+<img src="images/screenshot.gif" class="attempt-right" style="max-width: 300px">
+
+If you are new to TensorFlow Lite and are working with Android, we recommend
+exploring the following example application that can help you get started.
+
+<a class="button button-primary" href="https://github.com/tensorflow/examples/tree/master/lite/examples/reinforcement_learning/android">Android
+example</a>
+
+If you are using a platform other than Android, or you are already familiar with
+the [TensorFlow Lite APIs](https://www.tensorflow.org/api_docs/python/tf/lite),
+you can download our trained model.
+
+<a class="button button-primary" href="https://github.com/tensorflow/examples/blob/master/lite/examples/reinforcement_learning/android/app/src/main/assets/planestrike_tf.tflite">Download
+model</a>
+
+## How it works
+
+The model is built for a game agent to play a small board game called 'Plane
+Strike'. For a quick introduction of this game and its rules, please refer to
+this
+[README](https://github.com/tensorflow/examples/tree/master/lite/examples/reinforcement_learning/android).
+
+Underneath the app's UI, we have built an agent that plays against the human
+player. The agent is a 3-layer MLP that takes the board state as input and
+outputs the predicted score for each of the 64 possible board cells. The model
+is trained using policy gradient (REINFORCE) and you can find the training code
+[here](https://github.com/tensorflow/examples/blob/master/lite/examples/reinforcement_learning/ml).
+After training the agent, we convert the model into TFLite and deploy it in the
+Android app.
+
+During the actual game play in the Android app, when it's the agent's turn to
+take action, the agent looks at the human player's board state (the board at the
+bottom), which contains information about previous successful and unsuccessful
+strikes (hits and misses), and uses the trained model to predict where to strike
+next, so that it can finish the game before the human player does.
+
+## Performance benchmarks
+
+Performance benchmark numbers are generated with the tool described
+[here](https://www.tensorflow.org/lite/performance/benchmarks).
+
+<table>
+  <thead>
+    <tr>
+      <th>Model Name</th>
+      <th>Model size </th>
+      <th>Device </th>
+      <th>CPU</th>
+    </tr>
+  </thead>
+  <tr>
+    <td rowspan = 2>
+      <a href="https://github.com/tensorflow/examples/blob/master/lite/examples/reinforcement_learning/android/app/src/main/assets/planestrike.tflite">Policy Gradient</a>
+    </td>
+    <td rowspan = 2>
+      84 Kb
+    </td>
+    <td>Pixel 3 (Android 10) </td>
+    <td>0.01ms*</td>
+  </tr>
+   <tr>
+     <td>Pixel 4 (Android 10) </td>
+    <td>0.01ms*</td>
+  </tr>
+</table>
+
+\* 1 threads used.
+
+## Inputs
+
+The model accepts a 3-D `float32` Tensor of (1, 8, 8) as the board state.
+
+## Outputs
+
+The model returns a 2-D `float32` Tensor of shape (1,64) as the predicted scores
+for each of the 64 possible strike positions.
+
+## Train your own model
+
+You could train your own model for a larger/smaller board by change the
+`BOARD_SIZE` parameter in the
+[training code](https://github.com/tensorflow/examples/blob/master/lite/examples/reinforcement_learning/ml).

tensorflow/lite/g3doc/examples/segmentation/overview.md

@@ -0,0 +1,119 @@
+# Segmentation
+
+Image segmentation is the process of partitioning a digital image into multiple
+segments (sets of pixels, also known as image objects). The goal of segmentation
+is to simplify and/or change the representation of an image into something that
+is more meaningful and easier to analyze.
+
+The following image shows the output of the image segmentation model on Android.
+The model will create a mask over the target objects with high accuracy.
+
+<img src="images/segmentation.gif" class="attempt-right" />
+
+Note: To integrate an existing model, try
+[TensorFlow Lite Task Library](https://www.tensorflow.org/lite/inference_with_metadata/task_library/image_segmenter).
+
+## Get started
+
+If you are new to TensorFlow Lite and are working with Android or iOS, it is
+recommended you explore the following example applications that can help you get
+started.
+
+You can leverage the out-of-box API from
+[TensorFlow Lite Task Library](../../inference_with_metadata/task_library/image_segmenter)
+to integrate image segmentation models within just a few lines of code. You can
+also integrate the model using the
+[TensorFlow Lite Interpreter Java API](../../guide/inference#load_and_run_a_model_in_java).
+
+The Android example below demonstrates the implementation for both methods as
+[lib_task_api](https://github.com/tensorflow/examples/tree/master/lite/examples/image_segmentation/android/lib_task_api)
+and
+[lib_interpreter](https://github.com/tensorflow/examples/tree/master/lite/examples/image_segmentation/android/lib_interpreter),
+respectively.
+
+<a class="button button-primary" href="https://github.com/tensorflow/examples/tree/master/lite/examples/image_segmentation/android">View
+Android example</a>
+
+<a class="button button-primary" href="https://github.com/tensorflow/examples/tree/master/lite/examples/image_segmentation/ios">View
+iOS example</a>
+
+If you are using a platform other than Android or iOS, or you are already
+familiar with the
+<a href="https://www.tensorflow.org/api_docs/python/tf/lite">TensorFlow Lite
+APIs</a>, you can download our starter image segmentation model.
+
+<a class="button button-primary" href="https://tfhub.dev/tensorflow/lite-model/deeplabv3/1/metadata/2?lite-format=tflite">Download
+starter model</a>
+
+## Model description
+
+_DeepLab_ is a state-of-art deep learning model for semantic image segmentation,
+where the goal is to assign semantic labels (e.g. person, dog, cat) to every
+pixel in the input image.
+
+### How it works
+
+Semantic image segmentation predicts whether each pixel of an image is
+associated with a certain class. This is in contrast to
+<a href="../object_detection/overview.md">object detection</a>, which detects
+objects in rectangular regions, and
+<a href="../image_classification/overview.md">image classification</a>, which
+classifies the overall image.
+
+The current implementation includes the following features:
+<ol>
+  <li>DeepLabv1: We use atrous convolution to explicitly control the resolution at which feature responses are computed within Deep Convolutional Neural Networks.</li>
+  <li>DeepLabv2: We use atrous spatial pyramid pooling (ASPP) to robustly segment objects at multiple scales with filters at multiple sampling rates and effective fields-of-views.</li>
+  <li>DeepLabv3: We augment the ASPP module with image-level feature [5, 6] to capture longer range information. We also include batch normalization [7] parameters to facilitate the training. In particular, we applying atrous convolution to extract output features at different output strides during training and evaluation, which efficiently enables training BN at output stride = 16 and attains a high performance at output stride = 8 during evaluation.</li>
+  <li>DeepLabv3+: We extend DeepLabv3 to include a simple yet effective decoder module to refine the segmentation results especially along object boundaries. Furthermore, in this encoder-decoder structure one can arbitrarily control the resolution of extracted encoder features by atrous convolution to trade-off precision and runtime.</li>
+</ol>
+
+## Performance benchmarks
+
+Performance benchmark numbers are generated with the tool
+[described here](https://www.tensorflow.org/lite/performance/benchmarks).
+
+<table>
+  <thead>
+    <tr>
+      <th>Model Name</th>
+      <th>Model size </th>
+      <th>Device </th>
+      <th>GPU</th>
+      <th>CPU</th>
+    </tr>
+  </thead>
+  <tr>
+    <td rowspan = 3>
+      <a href="https://tfhub.dev/tensorflow/lite-model/deeplabv3/1/metadata/2?lite-format=tflite">Deeplab v3</a>
+    </td>
+    <td rowspan = 3>
+      2.7 Mb
+    </td>
+    <td>Pixel 3 (Android 10) </td>
+    <td>16ms</td>
+    <td>37ms*</td>
+  </tr>
+   <tr>
+     <td>Pixel 4 (Android 10) </td>
+    <td>20ms</td>
+    <td>23ms*</td>
+  </tr>
+   <tr>
+     <td>iPhone XS (iOS 12.4.1) </td>
+     <td>16ms</td>
+    <td>25ms** </td>
+  </tr>
+</table>
+
+\* 4 threads used.
+
+\*\* 2 threads used on iPhone for the best performance result.
+
+## Further reading and resources
+
+<ul>
+  <li><a href="https://ai.googleblog.com/2018/03/semantic-image-segmentation-with.html">Semantic Image Segmentation with DeepLab in TensorFlow</a></li>
+  <li><a href="https://medium.com/tensorflow/tensorflow-lite-now-faster-with-mobile-gpus-developer-preview-e15797e6dee7">TensorFlow Lite Now Faster with Mobile GPUs (Developer Preview)</a></li>
+  <li><a href="https://github.com/tensorflow/models/tree/master/research/deeplab">DeepLab: Deep Labelling for Semantic Image Segmentation</a></li>
+</ul>

tensorflow/lite/g3doc/examples/smart_reply/overview.md

@@ -0,0 +1,56 @@
+# Smart reply
+
+<img src="../images/smart_reply.png" class="attempt-right" />
+
+## Get started
+
+Our smart reply model generates reply suggestions based on chat messages. The
+suggestions are intended to be contextually relevant, one-touch responses that
+help the user to easily reply to an incoming message.
+
+<a class="button button-primary" href="https://tfhub.dev/tensorflow/lite-model/smartreply/1/default/1?lite-format=tflite">Download
+starter model</a>
+
+### Sample application
+
+There is a TensorFlow Lite sample application that demonstrates the smart reply
+model on Android.
+
+<a class="button button-primary" href="https://github.com/tensorflow/examples/tree/master/lite/examples/smart_reply/android">View
+Android example</a>
+
+Read the
+[GitHub page](https://github.com/tensorflow/examples/tree/master/lite/examples/smart_reply/android/)
+to learn how the app works. Inside this project, you will also learn how to
+build an app with custom C++ ops.
+
+## How it works
+
+The model generates reply suggestions to conversational chat messages.
+
+The on-device model comes with several benefits. It is:
+<ul>
+  <li>Fast: The model resides on the device and does not require internet connectivity. Thus, inference is very fast and has an average latency of only a few milliseconds.</li>
+  <li>Resource efficient: The model has a small memory footprint on the device.</li>
+  <li>Privacy-friendly: User data never leaves the device.</li>
+</ul>
+
+## Example output
+
+<img alt="Animation showing smart reply" src="images/smart_reply.gif" style="max-width: 300px"/>
+
+## Read more about this
+
+<ul>
+  <li><a href="https://arxiv.org/pdf/1708.00630.pdf">Research paper</a></li>
+  <li><a href="https://github.com/tensorflow/examples/tree/master/lite/examples/smart_reply/android">Source code</a></li>
+</ul>
+
+## Users
+
+<ul>
+  <li><a href="https://www.blog.google/products/gmail/save-time-with-smart-reply-in-gmail/">Gmail</a></li>
+  <li><a href="https://www.blog.google/products/gmail/computer-respond-to-this-email/">Inbox</a></li>
+  <li><a href="https://blog.google/products/allo/google-allo-smarter-messaging-app/">Allo</a></li>
+  <li><a href="https://research.googleblog.com/2017/02/on-device-machine-intelligence.html">Smart Replies on Android Wear</a></li>
+</ul>

tensorflow/lite/g3doc/examples/style_transfer/overview.ipynb

@@ -0,0 +1,480 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "g_nWetWWd_ns"
+      },
+      "source": [
+        "##### Copyright 2024 The AI Edge Authors."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "cellView": "form",
+        "id": "2pHVBk_seED1"
+      },
+      "outputs": [],
+      "source": [
+        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
+        "# you may not use this file except in compliance with the License.\n",
+        "# You may obtain a copy of the License at\n",
+        "#\n",
+        "# https://www.apache.org/licenses/LICENSE-2.0\n",
+        "#\n",
+        "# Unless required by applicable law or agreed to in writing, software\n",
+        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
+        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
+        "# See the License for the specific language governing permissions and\n",
+        "# limitations under the License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "M7vSdG6sAIQn"
+      },
+      "source": [
+        "# Artistic Style Transfer with TensorFlow Lite"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "fwc5GKHBASdc"
+      },
+      "source": [
+        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
+        "  <td>\n",
+        "    <a target=\"_blank\" href=\"https://www.tensorflow.org/lite/examples/style_transfer/overview\"><img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" />View on TensorFlow.org</a>\n",
+        "  </td>\n",
+        "  <td>\n",
+        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/examples/style_transfer/overview.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
+        "  </td>\n",
+        "  <td>\n",
+        "    <a target=\"_blank\" href=\"https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/examples/style_transfer/overview.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
+        "  </td>\n",
+        "  <td>\n",
+        "    <a href=\"https://storage.googleapis.com/tensorflow_docs/tensorflow/tensorflow/lite/g3doc/examples/style_transfer/overview.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\" />Download notebook</a>\n",
+        "  </td>\n",
+        "  <td>\n",
+        "    <a href=\"https://tfhub.dev/google/magenta/arbitrary-image-stylization-v1-256/2\"><img src=\"https://www.tensorflow.org/images/hub_logo_32px.png\" />See TF Hub model</a>\n",
+        "  </td>\n",
+        "</table>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "31O0iaROAw8z"
+      },
+      "source": [
+        "One of the most exciting developments in deep learning to come out recently is [artistic style transfer](https://arxiv.org/abs/1508.06576), or the ability to create a new image, known as a [pastiche](https://en.wikipedia.org/wiki/Pastiche), based on two input images: one representing the artistic style and one representing the content.\n",
+        "\n",
+        "![Style transfer example](https://storage.googleapis.com/download.tensorflow.org/models/tflite/arbitrary_style_transfer/formula.png)\n",
+        "\n",
+        "Using this technique, we can generate beautiful new artworks in a range of styles.\n",
+        "\n",
+        "![Style transfer example](https://storage.googleapis.com/download.tensorflow.org/models/tflite/arbitrary_style_transfer/table.png)\n",
+        "\n",
+        "If you are new to TensorFlow Lite and are working with Android, we\n",
+        "recommend exploring the following example applications that can help you get\n",
+        "started.\n",
+        "\n",
+        "<a class=\"button button-primary\" href=\"https://github.com/tensorflow/examples/tree/master/lite/examples/style_transfer/android\">Android\n",
+        "example</a> <a class=\"button button-primary\" href=\"https://github.com/tensorflow/examples/tree/master/lite/examples/style_transfer/ios\">iOS\n",
+        "example</a>\n",
+        "\n",
+        "If you are using a platform other than Android or iOS, or you are already\n",
+        "familiar with the\n",
+        "<a href=\"https://www.tensorflow.org/api_docs/python/tf/lite\">TensorFlow Lite\n",
+        "APIs</a>, you can follow this tutorial to learn how to apply style transfer on any pair of content and style image with a pre-trained TensorFlow Lite model. You can use the model to add style transfer to your own mobile applications.\n",
+        "\n",
+        "The model is open-sourced on [GitHub](https://github.com/tensorflow/magenta/tree/master/magenta/models/arbitrary_image_stylization#train-a-model-on-a-large-dataset-with-data-augmentation-to-run-on-mobile). You can retrain the model with different parameters (e.g. increase content layers' weights to make the output image look more like the content image)."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "ak0S4gkOCSxs"
+      },
+      "source": [
+        "## Understand the model architecture"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "oee6G_bBCgAM"
+      },
+      "source": [
+        "![Model Architecture](https://storage.googleapis.com/download.tensorflow.org/models/tflite/arbitrary_style_transfer/architecture.png)\n",
+        "\n",
+        "This Artistic Style Transfer model consists of two submodels:\n",
+        "1. **Style Prediciton Model**: A MobilenetV2-based neural network that takes an input style image to a 100-dimension style bottleneck vector.\n",
+        "1. **Style Transform Model**: A neural network that takes apply a style bottleneck vector to a content image and creates a stylized image.\n",
+        "\n",
+        "If your app only needs to support a fixed set of style images, you can compute their style bottleneck vectors in advance, and exclude the Style Prediction Model from your app's binary."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "a7ZETsRVNMo7"
+      },
+      "source": [
+        "## Setup"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "3n8oObKZN4c8"
+      },
+      "source": [
+        "Import dependencies."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "xz62Lb1oNm97"
+      },
+      "outputs": [],
+      "source": [
+        "import tensorflow as tf\n",
+        "print(tf.__version__)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "1Ua5FpcJNrIj"
+      },
+      "outputs": [],
+      "source": [
+        "import IPython.display as display\n",
+        "\n",
+        "import matplotlib.pyplot as plt\n",
+        "import matplotlib as mpl\n",
+        "mpl.rcParams['figure.figsize'] = (12,12)\n",
+        "mpl.rcParams['axes.grid'] = False\n",
+        "\n",
+        "import numpy as np\n",
+        "import time\n",
+        "import functools"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "1b988wrrQnVF"
+      },
+      "source": [
+        "Download the content and style images, and the pre-trained TensorFlow Lite models."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "16g57cIMQnen"
+      },
+      "outputs": [],
+      "source": [
+        "content_path = tf.keras.utils.get_file('belfry.jpg','https://storage.googleapis.com/khanhlvg-public.appspot.com/arbitrary-style-transfer/belfry-2611573_1280.jpg')\n",
+        "style_path = tf.keras.utils.get_file('style23.jpg','https://storage.googleapis.com/khanhlvg-public.appspot.com/arbitrary-style-transfer/style23.jpg')\n",
+        "\n",
+        "style_predict_path = tf.keras.utils.get_file('style_predict.tflite', 'https://tfhub.dev/google/lite-model/magenta/arbitrary-image-stylization-v1-256/int8/prediction/1?lite-format=tflite')\n",
+        "style_transform_path = tf.keras.utils.get_file('style_transform.tflite', 'https://tfhub.dev/google/lite-model/magenta/arbitrary-image-stylization-v1-256/int8/transfer/1?lite-format=tflite')"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "MQZXL7kON-gM"
+      },
+      "source": [
+        "## Pre-process the inputs\n",
+        "\n",
+        "* The content image and the style image must be RGB images with pixel values being float32 numbers between [0..1].\n",
+        "* The style image size must be (1, 256, 256, 3). We central crop the image and resize it.\n",
+        "* The content image must be (1, 384, 384, 3). We central crop the image and resize it."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "Cg0Vi-rXRUFl"
+      },
+      "outputs": [],
+      "source": [
+        "# Function to load an image from a file, and add a batch dimension.\n",
+        "def load_img(path_to_img):\n",
+        "  img = tf.io.read_file(path_to_img)\n",
+        "  img = tf.io.decode_image(img, channels=3)\n",
+        "  img = tf.image.convert_image_dtype(img, tf.float32)\n",
+        "  img = img[tf.newaxis, :]\n",
+        "\n",
+        "  return img\n",
+        "\n",
+        "# Function to pre-process by resizing an central cropping it.\n",
+        "def preprocess_image(image, target_dim):\n",
+        "  # Resize the image so that the shorter dimension becomes 256px.\n",
+        "  shape = tf.cast(tf.shape(image)[1:-1], tf.float32)\n",
+        "  short_dim = min(shape)\n",
+        "  scale = target_dim / short_dim\n",
+        "  new_shape = tf.cast(shape * scale, tf.int32)\n",
+        "  image = tf.image.resize(image, new_shape)\n",
+        "\n",
+        "  # Central crop the image.\n",
+        "  image = tf.image.resize_with_crop_or_pad(image, target_dim, target_dim)\n",
+        "\n",
+        "  return image\n",
+        "\n",
+        "# Load the input images.\n",
+        "content_image = load_img(content_path)\n",
+        "style_image = load_img(style_path)\n",
+        "\n",
+        "# Preprocess the input images.\n",
+        "preprocessed_content_image = preprocess_image(content_image, 384)\n",
+        "preprocessed_style_image = preprocess_image(style_image, 256)\n",
+        "\n",
+        "print('Style Image Shape:', preprocessed_style_image.shape)\n",
+        "print('Content Image Shape:', preprocessed_content_image.shape)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "xE4Yt8nArTeR"
+      },
+      "source": [
+        "## Visualize the inputs"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "ncPA4esJRcEu"
+      },
+      "outputs": [],
+      "source": [
+        "def imshow(image, title=None):\n",
+        "  if len(image.shape) > 3:\n",
+        "    image = tf.squeeze(image, axis=0)\n",
+        "\n",
+        "  plt.imshow(image)\n",
+        "  if title:\n",
+        "    plt.title(title)\n",
+        "\n",
+        "plt.subplot(1, 2, 1)\n",
+        "imshow(preprocessed_content_image, 'Content Image')\n",
+        "\n",
+        "plt.subplot(1, 2, 2)\n",
+        "imshow(preprocessed_style_image, 'Style Image')"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "CJ7R-CHbjC3s"
+      },
+      "source": [
+        "## Run style transfer with TensorFlow Lite"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "euu00ldHjKwD"
+      },
+      "source": [
+        "### Style prediction"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "o3zd9cTFRiS_"
+      },
+      "outputs": [],
+      "source": [
+        "# Function to run style prediction on preprocessed style image.\n",
+        "def run_style_predict(preprocessed_style_image):\n",
+        "  # Load the model.\n",
+        "  interpreter = tf.lite.Interpreter(model_path=style_predict_path)\n",
+        "\n",
+        "  # Set model input.\n",
+        "  interpreter.allocate_tensors()\n",
+        "  input_details = interpreter.get_input_details()\n",
+        "  interpreter.set_tensor(input_details[0][\"index\"], preprocessed_style_image)\n",
+        "\n",
+        "  # Calculate style bottleneck.\n",
+        "  interpreter.invoke()\n",
+        "  style_bottleneck = interpreter.tensor(\n",
+        "      interpreter.get_output_details()[0][\"index\"]\n",
+        "      )()\n",
+        "\n",
+        "  return style_bottleneck\n",
+        "\n",
+        "# Calculate style bottleneck for the preprocessed style image.\n",
+        "style_bottleneck = run_style_predict(preprocessed_style_image)\n",
+        "print('Style Bottleneck Shape:', style_bottleneck.shape)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "00t8S2PekIyW"
+      },
+      "source": [
+        "### Style transform"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "cZp5bCj8SX1w"
+      },
+      "outputs": [],
+      "source": [
+        "# Run style transform on preprocessed style image\n",
+        "def run_style_transform(style_bottleneck, preprocessed_content_image):\n",
+        "  # Load the model.\n",
+        "  interpreter = tf.lite.Interpreter(model_path=style_transform_path)\n",
+        "\n",
+        "  # Set model input.\n",
+        "  input_details = interpreter.get_input_details()\n",
+        "  interpreter.allocate_tensors()\n",
+        "\n",
+        "  # Set model inputs.\n",
+        "  interpreter.set_tensor(input_details[0][\"index\"], preprocessed_content_image)\n",
+        "  interpreter.set_tensor(input_details[1][\"index\"], style_bottleneck)\n",
+        "  interpreter.invoke()\n",
+        "\n",
+        "  # Transform content image.\n",
+        "  stylized_image = interpreter.tensor(\n",
+        "      interpreter.get_output_details()[0][\"index\"]\n",
+        "      )()\n",
+        "\n",
+        "  return stylized_image\n",
+        "\n",
+        "# Stylize the content image using the style bottleneck.\n",
+        "stylized_image = run_style_transform(style_bottleneck, preprocessed_content_image)\n",
+        "\n",
+        "# Visualize the output.\n",
+        "imshow(stylized_image, 'Stylized Image')"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "vv_71Td-QtrW"
+      },
+      "source": [
+        "### Style blending\n",
+        "\n",
+        "We can blend the style of content image into the stylized output, which in turn making the output look more like the content image."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "eJcAURXQQtJ7"
+      },
+      "outputs": [],
+      "source": [
+        "# Calculate style bottleneck of the content image.\n",
+        "style_bottleneck_content = run_style_predict(\n",
+        "    preprocess_image(content_image, 256)\n",
+        "    )"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "4S3yg2MgkmRD"
+      },
+      "outputs": [],
+      "source": [
+        "# Define content blending ratio between [0..1].\n",
+        "# 0.0: 0% style extracts from content image.\n",
+        "# 1.0: 100% style extracted from content image.\n",
+        "content_blending_ratio = 0.5 #@param {type:\"slider\", min:0, max:1, step:0.01}\n",
+        "\n",
+        "# Blend the style bottleneck of style image and content image\n",
+        "style_bottleneck_blended = content_blending_ratio * style_bottleneck_content \\\n",
+        "                           + (1 - content_blending_ratio) * style_bottleneck\n",
+        "\n",
+        "# Stylize the content image using the style bottleneck.\n",
+        "stylized_image_blended = run_style_transform(style_bottleneck_blended,\n",
+        "                                             preprocessed_content_image)\n",
+        "\n",
+        "# Visualize the output.\n",
+        "imshow(stylized_image_blended, 'Blended Stylized Image')"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "9k9jGIep8p1c"
+      },
+      "source": [
+        "## Performance Benchmarks\n",
+        "\n",
+        "Performance benchmark numbers are generated with the tool [described here](https://www.tensorflow.org/lite/performance/benchmarks).\n",
+        "<table ><thead><tr><th>Model name</th> <th>Model size</th>  <th>Device </th> <th>NNAPI</th> <th>CPU</th> <th>GPU</th></tr> </thead> \n",
+        "<tr> <td rowspan = 3> <a href=\"https://tfhub.dev/google/lite-model/magenta/arbitrary-image-stylization-v1-256/int8/prediction/1?lite-format=tflite\">Style prediction model (int8)</a> </td> \n",
+        "<td rowspan = 3>2.8 Mb</td>\n",
+        "<td>Pixel 3 (Android 10) </td> <td>142ms</td><td>14ms*</td><td></td></tr>\n",
+        "<tr><td>Pixel 4 (Android 10) </td> <td>5.2ms</td><td>6.7ms*</td><td></td></tr>\n",
+        "<tr><td>iPhone XS (iOS 12.4.1) </td> <td></td><td>10.7ms**</td><td></td></tr>\n",
+        "<tr> <td rowspan = 3> <a href=\"https://tfhub.dev/google/lite-model/magenta/arbitrary-image-stylization-v1-256/int8/transfer/1?lite-format=tflite\">Style transform model (int8)</a> </td> \n",
+        "<td rowspan = 3>0.2 Mb</td>\n",
+        "<td>Pixel 3 (Android 10) </td> <td></td><td>540ms*</td><td></td></tr>\n",
+        "<tr><td>Pixel 4 (Android 10) </td> <td></td><td>405ms*</td><td></td></tr>\n",
+        "<tr><td>iPhone XS (iOS 12.4.1) </td> <td></td><td>251ms**</td><td></td></tr>\n",
+        "\n",
+        "<tr> <td rowspan = 2> <a href=\"https://tfhub.dev/google/lite-model/magenta/arbitrary-image-stylization-v1-256/fp16/prediction/1?lite-format=tflite\">Style prediction model (float16)</a> </td> \n",
+        "<td rowspan = 2>4.7 Mb</td>\n",
+        "<td>Pixel 3 (Android 10) </td> <td>86ms</td><td>28ms*</td><td>9.1ms</td></tr>\n",
+        "<tr><td>Pixel 4 (Android 10) </td><td>32ms</td><td>12ms*</td><td>10ms</td></tr>\n",
+        "\n",
+        "<tr> <td rowspan = 2> <a href=\"https://tfhub.dev/google/lite-model/magenta/arbitrary-image-stylization-v1-256/fp16/transfer/1?lite-format=tflite\">Style transfer model (float16)</a> </td> \n",
+        "<td rowspan = 2>0.4 Mb</td>\n",
+        "<td>Pixel 3 (Android 10) </td> <td>1095ms</td><td>545ms*</td><td>42ms</td></tr>\n",
+        "<tr><td>Pixel 4 (Android 10) </td><td>603ms</td><td>377ms*</td><td>42ms</td></tr>\n",
+        "\n",
+        "</table>\n",
+        "\n",
+        "*&ast; 4 threads used. <br/>*\n",
+        "*&ast;&ast; 2 threads on iPhone for the best performance.*\n"
+      ]
+    }
+  ],
+  "metadata": {
+    "colab": {
+      "collapsed_sections": [],
+      "name": "overview.ipynb",
+      "provenance": [],
+      "toc_visible": true
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}

tensorflow/lite/g3doc/examples/super_resolution/overview.ipynb

@@ -0,0 +1,353 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "JfOIB1KdkbYW"
+      },
+      "source": [
+        "##### Copyright 2024 The AI Edge Authors."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "cellView": "form",
+        "id": "Ojb0aXCmBgo7"
+      },
+      "outputs": [],
+      "source": [
+        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
+        "# you may not use this file except in compliance with the License.\n",
+        "# You may obtain a copy of the License at\n",
+        "#\n",
+        "# https://www.apache.org/licenses/LICENSE-2.0\n",
+        "#\n",
+        "# Unless required by applicable law or agreed to in writing, software\n",
+        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
+        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
+        "# See the License for the specific language governing permissions and\n",
+        "# limitations under the License."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "M9Y4JZ0ZGoE4"
+      },
+      "source": [
+        "# Super resolution with TensorFlow Lite"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "q3FoFSLBjIYK"
+      },
+      "source": [
+        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
+        "  <td>\n",
+        "    <a target=\"_blank\" href=\"https://www.tensorflow.org/lite/examples/super_resolution/overview\"><img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" />View on TensorFlow.org</a>\n",
+        "  </td>\n",
+        "  <td>\n",
+        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/examples/super_resolution/overview.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
+        "  </td>\n",
+        "  <td>\n",
+        "    <a target=\"_blank\" href=\"https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/examples/super_resolution/overview.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
+        "  </td>\n",
+        "  <td>\n",
+        "    <a href=\"https://storage.googleapis.com/tensorflow_docs/tensorflow/tensorflow/lite/g3doc/examples/super_resolution/overview.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\" />Download notebook</a>\n",
+        "  </td>\n",
+        "  <td>\n",
+        "    <a href=\"https://tfhub.dev/captain-pool/esrgan-tf2/1\"><img src=\"https://www.tensorflow.org/images/hub_logo_32px.png\" />See TF Hub model</a>\n",
+        "  </td>\n",
+        "</table>"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "-uF3N4BbaMvA"
+      },
+      "source": [
+        "## Overview"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "isbXET4vVHfu"
+      },
+      "source": [
+        "The task of recovering a high resolution (HR) image from its low resolution counterpart is commonly referred to as Single Image Super Resolution (SISR). \n",
+        "\n",
+        "The model used here is ESRGAN\n",
+        "([ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks](https://arxiv.org/abs/1809.00219)). And we are going to use TensorFlow Lite to run inference on the pretrained model.\n",
+        "\n",
+        "The TFLite model is converted from this\n",
+        "[implementation](https://tfhub.dev/captain-pool/esrgan-tf2/1) hosted on TF Hub. Note that the model we converted upsamples a 50x50 low resolution image to a 200x200 high resolution image (scale factor=4). If you want a different input size or scale factor, you need to re-convert or re-train the original model."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "2dQlTqiffuoU"
+      },
+      "source": [
+        "## Setup"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "qKyMtsGqu3zH"
+      },
+      "source": [
+        "Let's install required libraries first."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "7YTT1Rxsw3A9"
+      },
+      "outputs": [],
+      "source": [
+        "!pip install matplotlib tensorflow tensorflow-hub"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "Clz5Kl97FswD"
+      },
+      "source": [
+        "Import dependencies."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "2xh1kvGEBjuP"
+      },
+      "outputs": [],
+      "source": [
+        "import tensorflow as tf\n",
+        "import tensorflow_hub as hub\n",
+        "import matplotlib.pyplot as plt\n",
+        "print(tf.__version__)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "i5miVfL4kxTA"
+      },
+      "source": [
+        "Download and convert the ESRGAN model"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "X5PvXIXRwvHj"
+      },
+      "outputs": [],
+      "source": [
+        "model = hub.load(\"https://tfhub.dev/captain-pool/esrgan-tf2/1\")\n",
+        "concrete_func = model.signatures[tf.saved_model.DEFAULT_SERVING_SIGNATURE_DEF_KEY]\n",
+        "\n",
+        "@tf.function(input_signature=[tf.TensorSpec(shape=[1, 50, 50, 3], dtype=tf.float32)])\n",
+        "def f(input):\n",
+        "  return concrete_func(input);\n",
+        "\n",
+        "converter = tf.lite.TFLiteConverter.from_concrete_functions([f.get_concrete_function()], model)\n",
+        "converter.optimizations = [tf.lite.Optimize.DEFAULT]\n",
+        "tflite_model = converter.convert()\n",
+        "\n",
+        "# Save the TF Lite model.\n",
+        "with tf.io.gfile.GFile('ESRGAN.tflite', 'wb') as f:\n",
+        "  f.write(tflite_model)\n",
+        "\n",
+        "esrgan_model_path = './ESRGAN.tflite'"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "jH5-xPkyUEqt"
+      },
+      "source": [
+        "Download a test image (insect head)."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "suWiStTWgK6e"
+      },
+      "outputs": [],
+      "source": [
+        "test_img_path = tf.keras.utils.get_file('lr.jpg', 'https://raw.githubusercontent.com/tensorflow/examples/master/lite/examples/super_resolution/android/app/src/main/assets/lr-1.jpg')"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "rgQ4qRuFNpyW"
+      },
+      "source": [
+        "## Generate a super resolution image using TensorFlow Lite"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "J9FV4btf02-2"
+      },
+      "outputs": [],
+      "source": [
+        "lr = tf.io.read_file(test_img_path)\n",
+        "lr = tf.image.decode_jpeg(lr)\n",
+        "lr = tf.expand_dims(lr, axis=0)\n",
+        "lr = tf.cast(lr, tf.float32)\n",
+        "\n",
+        "# Load TFLite model and allocate tensors.\n",
+        "interpreter = tf.lite.Interpreter(model_path=esrgan_model_path)\n",
+        "interpreter.allocate_tensors()\n",
+        "\n",
+        "# Get input and output tensors.\n",
+        "input_details = interpreter.get_input_details()\n",
+        "output_details = interpreter.get_output_details()\n",
+        "\n",
+        "# Run the model\n",
+        "interpreter.set_tensor(input_details[0]['index'], lr)\n",
+        "interpreter.invoke()\n",
+        "\n",
+        "# Extract the output and postprocess it\n",
+        "output_data = interpreter.get_tensor(output_details[0]['index'])\n",
+        "sr = tf.squeeze(output_data, axis=0)\n",
+        "sr = tf.clip_by_value(sr, 0, 255)\n",
+        "sr = tf.round(sr)\n",
+        "sr = tf.cast(sr, tf.uint8)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "EwddQrDUNQGO"
+      },
+      "source": [
+        "## Visualize the result"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "aasKuozt1gNd"
+      },
+      "outputs": [],
+      "source": [
+        "lr = tf.cast(tf.squeeze(lr, axis=0), tf.uint8)\n",
+        "plt.figure(figsize = (1, 1))\n",
+        "plt.title('LR')\n",
+        "plt.imshow(lr.numpy());\n",
+        "\n",
+        "plt.figure(figsize=(10, 4))\n",
+        "plt.subplot(1, 2, 1)        \n",
+        "plt.title(f'ESRGAN (x4)')\n",
+        "plt.imshow(sr.numpy());\n",
+        "\n",
+        "bicubic = tf.image.resize(lr, [200, 200], tf.image.ResizeMethod.BICUBIC)\n",
+        "bicubic = tf.cast(bicubic, tf.uint8)\n",
+        "plt.subplot(1, 2, 2)   \n",
+        "plt.title('Bicubic')\n",
+        "plt.imshow(bicubic.numpy());"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "0kb-fkogObjq"
+      },
+      "source": [
+        "## Performance Benchmarks"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "tNzdgpqTy5P3"
+      },
+      "source": [
+        "Performance benchmark numbers are generated with the tool\n",
+        "[described here](https://www.tensorflow.org/lite/performance/benchmarks).\n",
+        "\n",
+        "<table>\n",
+        "  <thead>\n",
+        "    <tr>\n",
+        "      <th>Model Name</th>\n",
+        "      <th>Model Size </th>\n",
+        "      <th>Device </th>\n",
+        "      <th>CPU</th>\n",
+        "      <th>GPU</th>\n",
+        "    </tr>\n",
+        "  </thead>\n",
+        "  <tr>\n",
+        "    <td rowspan = 3>\n",
+        "      super resolution (ESRGAN)\n",
+        "    </td>\n",
+        "    <td rowspan = 3>\n",
+        "      4.8 Mb\n",
+        "    </td>\n",
+        "    <td>Pixel 3</td>\n",
+        "    <td>586.8ms*</td>\n",
+        "      <td>128.6ms</td>\n",
+        "  </tr>\n",
+        "  <tr>\n",
+        "     <td>Pixel 4</td>\n",
+        "    <td>385.1ms*</td>\n",
+        "      <td>130.3ms</td>\n",
+        "  </tr>\n",
+        "\n",
+        "</table>\n",
+        "\n",
+        "**4 threads used*"
+      ]
+    }
+  ],
+  "metadata": {
+    "colab": {
+      "collapsed_sections": [],
+      "name": "super_resolution.ipynb",
+      "provenance": [],
+      "toc_visible": true
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "language": "python",
+      "name": "python3"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.7.4"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}

tensorflow/lite/g3doc/examples/text_classification/overview.md

@@ -0,0 +1,118 @@
+# Text classification
+
+Use a TensorFlow Lite model to category a paragraph into predefined groups.
+
+Note: (1) To integrate an existing model, try
+[TensorFlow Lite Task Library](https://www.tensorflow.org/lite/inference_with_metadata/task_library/nl_classifier).
+(2) To customize a model, try
+[TensorFlow Lite Model Maker](https://ai.google.dev/edge/litert/libraries/modify/text_classification).
+
+## Get started
+
+<img src="images/screenshot.gif" class="attempt-right" style="max-width: 300px">
+
+If you are new to TensorFlow Lite and are working with Android, we recommend
+exploring the guide of
+[TensorFLow Lite Task Library](../../inference_with_metadata/task_library/nl_classifier.md)
+to integrate text classification models within just a few lines of code. You can
+also integrate the model using the
+[TensorFlow Lite Interpreter Java API](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/guide/inference.md#load-and-run-a-model-in-java).
+
+The Android example below demonstrates the implementation for both methods as
+[lib_task_api](https://github.com/tensorflow/examples/tree/master/lite/examples/text_classification/android/lib_task_api)
+and
+[lib_interpreter](https://github.com/tensorflow/examples/tree/master/lite/examples/text_classification/android/lib_interpreter),
+respectively.
+
+<a class="button button-primary" href="https://github.com/tensorflow/examples/tree/master/lite/examples/text_classification/android">Android
+example</a>
+
+If you are using a platform other than Android, or you are already familiar with
+the TensorFlow Lite APIs, you can download our starter text classification
+model.
+
+<a class="button button-primary" href="https://storage.googleapis.com/download.tensorflow.org/models/tflite/text_classification/text_classification_v2.tflite">Download
+starter model</a>
+
+## How it works
+
+Text classification categorizes a paragraph into predefined groups based on its
+content.
+
+This pretrained model predicts if a paragraph's sentiment is positive or
+negative. It was trained on
+[Large Movie Review Dataset v1.0](http://ai.stanford.edu/~amaas/data/sentiment/)
+from Mass et al, which consists of IMDB movie reviews labeled as either positive
+or negative.
+
+Here are the steps to classify a paragraph with the model:
+
+1.  Tokenize the paragraph and convert it to a list of word ids using a
+    predefined vocabulary.
+1.  Feed the list to the TensorFlow Lite model.
+1.  Get the probability of the paragraph being positive or negative from the
+    model outputs.
+
+### Note
+
+*   Only English is supported.
+*   This model was trained on movie reviews dataset so you may experience
+    reduced accuracy when classifying text of other domains.
+
+## Performance benchmarks
+
+Performance benchmark numbers are generated with the tool
+[described here](https://www.tensorflow.org/lite/performance/benchmarks).
+
+<table>
+  <thead>
+    <tr>
+      <th>Model Name</th>
+      <th>Model size </th>
+      <th>Device </th>
+      <th>CPU</th>
+    </tr>
+  </thead>
+  <tr>
+    <td rowspan = 3>
+      <a href="https://storage.googleapis.com/download.tensorflow.org/models/tflite/text_classification/text_classification_v2.tflite">Text Classification</a>
+    </td>
+    <td rowspan = 3>
+      0.6 Mb
+    </td>
+    <td>Pixel 3 (Android 10) </td>
+    <td>0.05ms*</td>
+  </tr>
+   <tr>
+     <td>Pixel 4 (Android 10) </td>
+    <td>0.05ms*</td>
+  </tr>
+   <tr>
+     <td>iPhone XS (iOS 12.4.1) </td>
+    <td>0.025ms** </td>
+  </tr>
+</table>
+
+\* 4 threads used.
+
+\*\* 2 threads used on iPhone for the best performance result.
+
+## Example output
+
+| Text                                       | Negative (0) | Positive (1) |
+| ------------------------------------------ | ------------ | ------------ |
+| This is the best movie I’ve seen in recent | 25.3%        | 74.7%        |
+: years. Strongly recommend it!              :              :              :
+| What a waste of my time.                   | 72.5%        | 27.5%        |
+
+## Use your training dataset
+
+Follow this
+[tutorial](https://ai.google.dev/edge/litert/libraries/modify/text_classification)
+to apply the same technique used here to train a text classification model using
+your own datasets. With the right dataset, you can create a model for use cases
+such as document categorization or toxic comments detection.
+
+## Read more about text classification
+
+*   [Word embeddings and tutorial to train this model](https://www.tensorflow.org/tutorials/text/word_embeddings)

tensorflow/lite/g3doc/examples/video_classification/overview.md

@@ -0,0 +1,263 @@
+# Video classification
+
+<img src="../images/video.png" class="attempt-right">
+
+*Video classification* is the machine learning task of identifying what a video
+represents. A video classification model is trained on a video dataset that
+contains a set of unique classes, such as different actions or movements. The
+model receives video frames as input and outputs the probability of each class
+being represented in the video.
+
+Video classification and image classification models both use images as inputs
+to predict the probabilities of those images belonging to predefined classes.
+However, a video classification model also processes the spatio-temporal
+relationships between adjacent frames to recognize the actions in a video.
+
+For example, a *video action recognition* model can be trained to identify human
+actions like running, clapping, and waving. The following image shows the output
+of a video classification model on Android.
+
+<img alt="Screenshot of Android example" src="https://storage.googleapis.com/download.tensorflow.org/models/tflite/screenshots/push-up-classification.gif"/>
+
+## Get started
+
+If you are using a platform other than Android or Raspberry Pi, or if you are
+already familiar with the
+[TensorFlow Lite APIs](https://www.tensorflow.org/api_docs/python/tf/lite),
+download the starter video classification model and the supporting files. You
+can also build your own custom inference pipeline using the
+[TensorFlow Lite Support Library](../../inference_with_metadata/lite_support.md).
+
+<a class="button button-primary" href="https://tfhub.dev/tensorflow/lite-model/movinet/a0/stream/kinetics-600/classification/tflite/int8/1">Download
+starter model with metadata</a>
+
+If you are new to TensorFlow Lite and are working with Android or Raspberry Pi,
+explore the following example applications to help you get started.
+
+### Android
+
+The Android application uses the device's back camera for continuous video
+classification. Inference is performed using the
+[TensorFlow Lite Java API](https://www.tensorflow.org/lite/api_docs/java/org/tensorflow/lite/package-summary).
+The demo app classifies frames and displays the predicted classifications in
+real time.
+
+<a class="button button-primary" href="https://github.com/tensorflow/examples/tree/master/lite/examples/video_classification/android">Android
+example</a>
+
+### Raspberry Pi
+
+The Raspberry Pi example uses TensorFlow Lite with Python to perform continuous
+video classification. Connect the Raspberry Pi to a camera, like Pi Camera, to
+perform real-time video classification. To view results from the camera, connect
+a monitor to the Raspberry Pi and use SSH to access the Pi shell (to avoid
+connecting a keyboard to the Pi).
+
+Before starting,
+[set up](https://projects.raspberrypi.org/en/projects/raspberry-pi-setting-up)
+your Raspberry Pi with Raspberry Pi OS (preferably updated to Buster).
+
+<a class="button button-primary" href="https://github.com/tensorflow/examples/tree/master/lite/examples/video_classification/raspberry_pi ">Raspberry
+Pi example</a>
+
+## Model description
+
+Mobile Video Networks
+([MoViNets](https://github.com/tensorflow/models/tree/master/official/projects/movinet))
+are a family of efficient video classification models optimized for mobile
+devices. MoViNets demonstrate state-of-the-art accuracy and efficiency on
+several large-scale video action recognition datasets, making them well-suited
+for *video action recognition* tasks.
+
+There are three variants of the
+[MoviNet](https://tfhub.dev/s?deployment-format=lite&q=movinet) model for
+TensorFlow Lite:
+[MoviNet-A0](https://tfhub.dev/tensorflow/movinet/a0/stream/kinetics-600/classification),
+[MoviNet-A1](https://tfhub.dev/tensorflow/movinet/a1/stream/kinetics-600/classification),
+and
+[MoviNet-A2](https://tfhub.dev/tensorflow/movinet/a2/stream/kinetics-600/classification).
+These variants were trained with the
+[Kinetics-600](https://arxiv.org/abs/1808.01340) dataset to recognize 600
+different human actions. *MoviNet-A0* is the smallest, fastest, and least
+accurate. *MoviNet-A2* is the largest, slowest, and most accurate. *MoviNet-A1*
+is a compromise between A0 and A2.
+
+### How it works
+
+During training, a video classification model is provided videos and their
+associated *labels*. Each label is the name of a distinct concept, or class,
+that the model will learn to recognize. For *video action recognition*, the
+videos will be of human actions and the labels will be the associated action.
+
+The video classification model can learn to predict whether new videos belong to
+any of the classes provided during training. This process is called *inference*.
+You can also use
+[transfer learning](https://colab.research.google.com/github/tensorflow/models/blob/master/official/projects/movinet/movinet_tutorial.ipynb)
+to identify new classes of videos by using a pre-existing model.
+
+The model is a streaming model that receives continuous video and responds in
+real time. As the model receives a video stream, it identifies whether any of
+the classes from the training dataset are represented in the video. For each
+frame, the model returns these classes, along with the probability that the
+video represents the class. An example output at a given time might look as
+follows:
+
+<table style="width: 40%;">
+  <thead>
+    <tr>
+      <th>Action</th>
+      <th>Probability</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td>square dancing</td>
+      <td>0.02</td>
+    </tr>
+    <tr>
+      <td>threading needle</td>
+      <td>0.08</td>
+    </tr>
+    <tr>
+      <td>twiddling fingers</td>
+      <td>0.23</td>
+    </tr>
+    <tr>
+      <td style="background-color: #fcb66d;">Waving hand</td>
+      <td style="background-color: #fcb66d;">0.67</td>
+    </tr>
+  </tbody>
+</table>
+
+Each action in the output corresponds to a label in the training data. The
+probability denotes the likelihood that the action is being displayed in the
+video.
+
+### Model inputs
+
+The model accepts a stream of RGB video frames as input. The size of the input
+video is flexible, but ideally it matches the model training resolution and
+frame-rate:
+
+*   **MoviNet-A0**: 172 x 172 at 5 fps
+*   **MoviNet-A1**: 172 x 172 at 5 fps
+*   **MoviNet-A1**: 224 x 224 at 5 fps
+
+The input videos are expected to have color values within the range of 0 and 1,
+following the common
+[image input conventions](https://www.tensorflow.org/hub/common_signatures/images#input).
+
+Internally, the model also analyzes the context of each frame by using
+information gathered in previous frames. This is accomplished by taking internal
+states from the model output and feeding it back into the model for upcoming
+frames.
+
+### Model outputs
+
+The model returns a series of labels and their corresponding scores. The scores
+are logit values that represent the prediction for each class. These scores can
+be converted to probabilities by using the softmax function (`tf.nn.softmax`).
+
+```python
+    exp_logits = np.exp(np.squeeze(logits, axis=0))
+    probabilities = exp_logits / np.sum(exp_logits)
+```
+
+Internally, the model output also includes internal states from the model and
+feeds it back into the model for upcoming frames.
+
+## Performance benchmarks
+
+Performance benchmark numbers are generated with the
+[benchmarking tool](https://www.tensorflow.org/lite/performance/measurement).
+MoviNets only support CPU.
+
+Model performance is measured by the amount of time it takes for a model to run
+inference on a given piece of hardware. A lower time implies a faster model.
+Accuracy is measured by how often the model correctly classifies a class in a
+video.
+
+<table>
+  <thead>
+    <tr>
+      <th>Model Name</th>
+      <th>Size </th>
+      <th>Accuracy *</th>
+      <th>Device</th>
+      <th>CPU **</th>
+    </tr>
+  </thead>
+  <tr>
+    <td rowspan = 2>
+MoviNet-A0 (Integer quantized)
+    </td>
+    <td rowspan = 2>
+      3.1 MB
+    </td>
+    <td rowspan = 2>65%</td>
+    <td>Pixel 4</td>
+    <td>5 ms</td>
+  </tr>
+   <tr>
+    <td>Pixel 3</td>
+    <td>11 ms</td>
+  </tr>
+    <tr>
+    <td rowspan = 2>
+MoviNet-A1 (Integer quantized)
+    </td>
+    <td rowspan = 2>
+      4.5 MB
+    </td>
+    <td rowspan = 2>70%</td>
+    <td>Pixel 4</td>
+    <td>8 ms</td>
+  </tr>
+   <tr>
+    <td>Pixel 3</td>
+    <td>19 ms</td>
+  </tr>
+      <tr>
+    <td rowspan = 2>
+MoviNet-A2 (Integer quantized)
+    </td>
+    <td rowspan = 2>
+      5.1 MB
+    </td>
+    <td rowspan = 2>72%</td>
+    <td>Pixel 4</td>
+    <td>15 ms</td>
+  </tr>
+   <tr>
+    <td>Pixel 3</td>
+    <td>36 ms</td>
+  </tr>
+</table>
+
+\* Top-1 accuracy measured on the
+[Kinetics-600](https://arxiv.org/abs/1808.01340) dataset.
+
+\*\* Latency measured when running on CPU with 1-thread.
+
+## Model customization
+
+The pre-trained models are trained to recognize 600 human actions from the
+[Kinetics-600](https://arxiv.org/abs/1808.01340) dataset. You can also use
+transfer learning to re-train a model to recognize human actions that are not in
+the original set. To do this, you need a set of training videos for each of the
+new actions you want to incorporate into the model.
+
+For more on fine-tuning models on custom data, see the
+[MoViNets repo](https://github.com/tensorflow/models/tree/master/official/projects/movinet)
+and
+[MoViNets tutorial](https://colab.research.google.com/github/tensorflow/models/blob/master/official/projects/movinet/movinet_tutorial.ipynb).
+
+## Further reading and resources
+
+Use the following resources to learn more about concepts discussed on this page:
+
+*   [MoViNets repo](https://github.com/tensorflow/models/tree/master/official/projects/movinet)
+*   [MoViNets paper](https://arxiv.org/abs/2103.11511)
+*   [Pretrained MoViNet models](https://tfhub.dev/s?deployment-format=lite&q=movinet)
+*   [MoViNets tutorial](https://colab.research.google.com/github/tensorflow/models/blob/master/official/projects/movinet/movinet_tutorial.ipynb)
+*   [Kinetics datasets](https://deepmind.com/research/open-source/kinetics)