{ "cells": [ { "cell_type": "markdown", "id": "g_nWetWWd_ns", "metadata": { "id": "g_nWetWWd_ns" }, "source": [ "##### Copyright 2024 The AI Edge Authors." ] }, { "cell_type": "code", "execution_count": null, "id": "2pHVBk_seED1", "metadata": { "cellView": "form", "id": "2pHVBk_seED1" }, "outputs": [], "source": [ "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n", "# you may not use this file except in compliance with the License.\n", "# You may obtain a copy of the License at\n", "#\n", "# https://www.apache.org/licenses/LICENSE-2.0\n", "#\n", "# Unless required by applicable law or agreed to in writing, software\n", "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", "# See the License for the specific language governing permissions and\n", "# limitations under the License." ] }, { "cell_type": "markdown", "id": "M7vSdG6sAIQn", "metadata": { "id": "M7vSdG6sAIQn" }, "source": [ "# Signatures in TensorFlow Lite" ] }, { "cell_type": "markdown", "id": "fwc5GKHBASdc", "metadata": { "id": "fwc5GKHBASdc" }, "source": [ "\n", " \n", " \n", " \n", " \n", "
\n", " View on TensorFlow.org\n", " \n", " Run in Google Colab\n", " \n", " View source on GitHub\n", " \n", " Download notebook\n", "
" ] }, { "cell_type": "markdown", "id": "9ee074e4", "metadata": { "id": "9ee074e4" }, "source": [ "TensorFlow Lite supports converting TensorFlow model's input/output\n", "specifications to TensorFlow Lite models. The input/output specifications are\n", "called \"signatures\". Signatures can be specified when building a SavedModel or\n", "creating concrete functions.\n", "\n", "Signatures in TensorFlow Lite provide the following features:\n", "\n", "* They specify inputs and outputs of the converted TensorFlow Lite model by\n", " respecting the TensorFlow model's signatures.\n", "* Allow a single TensorFlow Lite model to support multiple entry points.\n", "\n", "The signature is composed of three pieces:\n", "\n", "* Inputs: Map for inputs from input name in the signature to an input tensor.\n", "* Outputs: Map for output mapping from output name in signature to an output\n", " tensor.\n", "* Signature Key: Name that identifies an entry point of the graph.\n", "\n" ] }, { "cell_type": "markdown", "id": "UaWdLA3fQDK2", "metadata": { "id": "UaWdLA3fQDK2" }, "source": [ "## Setup" ] }, { "cell_type": "code", "execution_count": null, "id": "9j4MGqyKQEo4", "metadata": { "id": "9j4MGqyKQEo4" }, "outputs": [], "source": [ "import tensorflow as tf" ] }, { "cell_type": "markdown", "id": "FN2N6hPEP-Ay", "metadata": { "id": "FN2N6hPEP-Ay" }, "source": [ "## Example model\n", "\n", "Let's say we have two tasks, e.g., encoding and decoding, as a TensorFlow model:" ] }, { "cell_type": "code", "execution_count": null, "id": "d8577c80", "metadata": { "id": "d8577c80" }, "outputs": [], "source": [ "class Model(tf.Module):\n", "\n", " @tf.function(input_signature=[tf.TensorSpec(shape=[None], dtype=tf.float32)])\n", " def encode(self, x):\n", " result = tf.strings.as_string(x)\n", " return {\n", " \"encoded_result\": result\n", " }\n", "\n", " @tf.function(input_signature=[tf.TensorSpec(shape=[None], dtype=tf.string)])\n", " def decode(self, x):\n", " result = tf.strings.to_number(x)\n", " return {\n", " \"decoded_result\": result\n", " }" ] }, { "cell_type": "markdown", "id": "9c814c6e", "metadata": { "id": "9c814c6e" }, "source": [ "In the signature wise, the above TensorFlow model can be summarized as follows:\n", "\n", "* Signature\n", "\n", " - Key: encode\n", " - Inputs: {\"x\"}\n", " - Output: {\"encoded_result\"}\n", "\n", "* Signature\n", "\n", " - Key: decode\n", " - Inputs: {\"x\"}\n", " - Output: {\"decoded_result\"}" ] }, { "cell_type": "markdown", "id": "c4099f20", "metadata": { "id": "c4099f20" }, "source": [ "## Convert a model with Signatures\n", "\n", "TensorFlow Lite converter APIs will bring the above signature information into\n", "the converted TensorFlow Lite model.\n", "\n", "This conversion functionality is available on all the converter APIs starting\n", "from TensorFlow version 2.7.0. See example usages.\n" ] }, { "cell_type": "markdown", "id": "Qv0WwFQkQgnO", "metadata": { "id": "Qv0WwFQkQgnO" }, "source": [ "\n", "### From Saved Model" ] }, { "cell_type": "code", "execution_count": null, "id": "96c8fc79", "metadata": { "id": "96c8fc79" }, "outputs": [], "source": [ "model = Model()\n", "\n", "# Save the model\n", "SAVED_MODEL_PATH = 'content/saved_models/coding'\n", "\n", "tf.saved_model.save(\n", " model, SAVED_MODEL_PATH,\n", " signatures={\n", " 'encode': model.encode.get_concrete_function(),\n", " 'decode': model.decode.get_concrete_function()\n", " })\n", "\n", "# Convert the saved model using TFLiteConverter\n", "converter = tf.lite.TFLiteConverter.from_saved_model(SAVED_MODEL_PATH)\n", "converter.target_spec.supported_ops = [\n", " tf.lite.OpsSet.TFLITE_BUILTINS, # enable TensorFlow Lite ops.\n", " tf.lite.OpsSet.SELECT_TF_OPS # enable TensorFlow ops.\n", "]\n", "tflite_model = converter.convert()\n", "\n", "# Print the signatures from the converted model\n", "interpreter = tf.lite.Interpreter(model_content=tflite_model)\n", "signatures = interpreter.get_signature_list()\n", "print(signatures)" ] }, { "cell_type": "markdown", "id": "5baa9f17", "metadata": { "id": "5baa9f17" }, "source": [ "### From Keras Model" ] }, { "cell_type": "code", "execution_count": null, "id": "71f29229", "metadata": { "id": "71f29229" }, "outputs": [], "source": [ "# Generate a Keras model.\n", "keras_model = tf.keras.Sequential(\n", " [\n", " tf.keras.layers.Dense(2, input_dim=4, activation='relu', name='x'),\n", " tf.keras.layers.Dense(1, activation='relu', name='output'),\n", " ]\n", ")\n", "\n", "# Convert the keras model using TFLiteConverter.\n", "# Keras model converter API uses the default signature automatically.\n", "converter = tf.lite.TFLiteConverter.from_keras_model(keras_model)\n", "tflite_model = converter.convert()\n", "\n", "# Print the signatures from the converted model\n", "interpreter = tf.lite.Interpreter(model_content=tflite_model)\n", "\n", "signatures = interpreter.get_signature_list()\n", "print(signatures)" ] }, { "cell_type": "markdown", "id": "e4d30f85", "metadata": { "id": "e4d30f85" }, "source": [ "### From Concrete Functions" ] }, { "cell_type": "code", "execution_count": null, "id": "c9e8a742", "metadata": { "id": "c9e8a742" }, "outputs": [], "source": [ "model = Model()\n", "\n", "# Convert the concrete functions using TFLiteConverter\n", "converter = tf.lite.TFLiteConverter.from_concrete_functions(\n", " [model.encode.get_concrete_function(),\n", " model.decode.get_concrete_function()], model)\n", "converter.target_spec.supported_ops = [\n", " tf.lite.OpsSet.TFLITE_BUILTINS, # enable TensorFlow Lite ops.\n", " tf.lite.OpsSet.SELECT_TF_OPS # enable TensorFlow ops.\n", "]\n", "tflite_model = converter.convert()\n", "\n", "# Print the signatures from the converted model\n", "interpreter = tf.lite.Interpreter(model_content=tflite_model)\n", "signatures = interpreter.get_signature_list()\n", "print(signatures)" ] }, { "cell_type": "markdown", "id": "b5e85934", "metadata": { "id": "b5e85934" }, "source": [ "## Run Signatures\n", "\n", "TensorFlow inference APIs support the signature-based executions:\n", "\n", "* Accessing the input/output tensors through the names of the inputs and\n", " outputs, specified by the signature.\n", "* Running each entry point of the graph separately, identified by the\n", " signature key.\n", "* Support for the SavedModel's initialization procedure.\n", "\n", "Java, C++ and Python language bindings are currently available. See example the\n", "below sections.\n" ] }, { "cell_type": "markdown", "id": "ZRBMFciMQmiB", "metadata": { "id": "ZRBMFciMQmiB" }, "source": [ "\n", "### Java" ] }, { "cell_type": "markdown", "id": "04c5a4fc", "metadata": { "id": "04c5a4fc" }, "source": [ "```\n", "try (Interpreter interpreter = new Interpreter(file_of_tensorflowlite_model)) {\n", " // Run encoding signature.\n", " Map inputs = new HashMap<>();\n", " inputs.put(\"x\", input);\n", " Map outputs = new HashMap<>();\n", " outputs.put(\"encoded_result\", encoded_result);\n", " interpreter.runSignature(inputs, outputs, \"encode\");\n", "\n", " // Run decoding signature.\n", " Map inputs = new HashMap<>();\n", " inputs.put(\"x\", encoded_result);\n", " Map outputs = new HashMap<>();\n", " outputs.put(\"decoded_result\", decoded_result);\n", " interpreter.runSignature(inputs, outputs, \"decode\");\n", "}\n", "```" ] }, { "cell_type": "markdown", "id": "5ba86c64", "metadata": { "id": "5ba86c64" }, "source": [ "### C++" ] }, { "cell_type": "markdown", "id": "397ad6fd", "metadata": { "id": "397ad6fd" }, "source": [ "```\n", "SignatureRunner* encode_runner =\n", " interpreter->GetSignatureRunner(\"encode\");\n", "encode_runner->ResizeInputTensor(\"x\", {100});\n", "encode_runner->AllocateTensors();\n", "\n", "TfLiteTensor* input_tensor = encode_runner->input_tensor(\"x\");\n", "float* input = GetTensorData(input_tensor);\n", "// Fill `input`.\n", "\n", "encode_runner->Invoke();\n", "\n", "const TfLiteTensor* output_tensor = encode_runner->output_tensor(\n", " \"encoded_result\");\n", "float* output = GetTensorData(output_tensor);\n", "// Access `output`.\n", "```" ] }, { "cell_type": "markdown", "id": "0f4c6ad4", "metadata": { "id": "0f4c6ad4" }, "source": [ "### Python" ] }, { "cell_type": "code", "execution_count": null, "id": "ab7b1963", "metadata": { "id": "ab7b1963" }, "outputs": [], "source": [ "# Load the TFLite model in TFLite Interpreter\n", "interpreter = tf.lite.Interpreter(model_content=tflite_model)\n", "\n", "# Print the signatures from the converted model\n", "signatures = interpreter.get_signature_list()\n", "print('Signature:', signatures)\n", "\n", "# encode and decode are callable with input as arguments.\n", "encode = interpreter.get_signature_runner('encode')\n", "decode = interpreter.get_signature_runner('decode')\n", "\n", "# 'encoded' and 'decoded' are dictionaries with all outputs from the inference.\n", "input = tf.constant([1, 2, 3], dtype=tf.float32)\n", "print('Input:', input)\n", "encoded = encode(x=input)\n", "print('Encoded result:', encoded)\n", "decoded = decode(x=encoded['encoded_result'])\n", "print('Decoded result:', decoded)" ] }, { "cell_type": "markdown", "id": "81b42e5b", "metadata": { "id": "81b42e5b" }, "source": [ "## Known limitations\n", "\n", "* As TFLite interpreter does not gurantee thread safety, the signature runners\n", " from the same interpreter won't be executed concurrently.\n", "* Support for iOS/Swift is not available yet.\n", "\n" ] }, { "cell_type": "markdown", "id": "3032Iof6QqmJ", "metadata": { "id": "3032Iof6QqmJ" }, "source": [ "## Updates\n", "\n", "* Version 2.7\n", " - The multiple signature feature is implemented.\n", " - All the converter APIs from version two generate signature-enabled\n", " TensorFlow Lite models.\n", "* Version 2.5\n", " - Signature feature is available through the `from_saved_model` converter\n", " API." ] } ], "metadata": { "colab": { "collapsed_sections": [], "name": "Signatures in TensorFlow Lite", "provenance": [], "toc_visible": true }, "id": "a1b42e5b", "kernelspec": { "display_name": "Python 3", "name": "python3" }, "language_info": { "name": "python" } }, "nbformat": 4, "nbformat_minor": 5 }