wehub/tensorflow--tensorflow
1
0
Fork 0
Code Issues Pull Requests Actions 5 Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution
cffconvert / validate (push) Has been skipped
Details
License Check / license-check (push) Failing after 2s
Details

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 12:14:16 +08:00
commit 8a852e4b4e
36502 changed files with 9277225 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tensorflow/lite/g3doc/examples/style_transfer/overview.ipynb
+480
View File
@@ -0,0 +1,480 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "g_nWetWWd_ns"
},
"source": [
"##### Copyright 2024 The AI Edge Authors."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "2pHVBk_seED1"
},
"outputs": [],
"source": [
"#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "M7vSdG6sAIQn"
},
"source": [
"# Artistic Style Transfer with TensorFlow Lite"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "fwc5GKHBASdc"
},
"source": [
"<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
" <td>\n",
" <a target=\"_blank\" href=\"https://www.tensorflow.org/lite/examples/style_transfer/overview\"><img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" />View on TensorFlow.org</a>\n",
" </td>\n",
" <td>\n",
" <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/examples/style_transfer/overview.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
" </td>\n",
" <td>\n",
" <a target=\"_blank\" href=\"https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/examples/style_transfer/overview.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://storage.googleapis.com/tensorflow_docs/tensorflow/tensorflow/lite/g3doc/examples/style_transfer/overview.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\" />Download notebook</a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://tfhub.dev/google/magenta/arbitrary-image-stylization-v1-256/2\"><img src=\"https://www.tensorflow.org/images/hub_logo_32px.png\" />See TF Hub model</a>\n",
" </td>\n",
"</table>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "31O0iaROAw8z"
},
"source": [
"One of the most exciting developments in deep learning to come out recently is [artistic style transfer](https://arxiv.org/abs/1508.06576), or the ability to create a new image, known as a [pastiche](https://en.wikipedia.org/wiki/Pastiche), based on two input images: one representing the artistic style and one representing the content.\n",
"\n",
"![Style transfer example](https://storage.googleapis.com/download.tensorflow.org/models/tflite/arbitrary_style_transfer/formula.png)\n",
"\n",
"Using this technique, we can generate beautiful new artworks in a range of styles.\n",
"\n",
"![Style transfer example](https://storage.googleapis.com/download.tensorflow.org/models/tflite/arbitrary_style_transfer/table.png)\n",
"\n",
"If you are new to TensorFlow Lite and are working with Android, we\n",
"recommend exploring the following example applications that can help you get\n",
"started.\n",
"\n",
"<a class=\"button button-primary\" href=\"https://github.com/tensorflow/examples/tree/master/lite/examples/style_transfer/android\">Android\n",
"example</a> <a class=\"button button-primary\" href=\"https://github.com/tensorflow/examples/tree/master/lite/examples/style_transfer/ios\">iOS\n",
"example</a>\n",
"\n",
"If you are using a platform other than Android or iOS, or you are already\n",
"familiar with the\n",
"<a href=\"https://www.tensorflow.org/api_docs/python/tf/lite\">TensorFlow Lite\n",
"APIs</a>, you can follow this tutorial to learn how to apply style transfer on any pair of content and style image with a pre-trained TensorFlow Lite model. You can use the model to add style transfer to your own mobile applications.\n",
"\n",
"The model is open-sourced on [GitHub](https://github.com/tensorflow/magenta/tree/master/magenta/models/arbitrary_image_stylization#train-a-model-on-a-large-dataset-with-data-augmentation-to-run-on-mobile). You can retrain the model with different parameters (e.g. increase content layers' weights to make the output image look more like the content image)."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ak0S4gkOCSxs"
},
"source": [
"## Understand the model architecture"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "oee6G_bBCgAM"
},
"source": [
"![Model Architecture](https://storage.googleapis.com/download.tensorflow.org/models/tflite/arbitrary_style_transfer/architecture.png)\n",
"\n",
"This Artistic Style Transfer model consists of two submodels:\n",
"1. **Style Prediciton Model**: A MobilenetV2-based neural network that takes an input style image to a 100-dimension style bottleneck vector.\n",
"1. **Style Transform Model**: A neural network that takes apply a style bottleneck vector to a content image and creates a stylized image.\n",
"\n",
"If your app only needs to support a fixed set of style images, you can compute their style bottleneck vectors in advance, and exclude the Style Prediction Model from your app's binary."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "a7ZETsRVNMo7"
},
"source": [
"## Setup"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "3n8oObKZN4c8"
},
"source": [
"Import dependencies."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "xz62Lb1oNm97"
},
"outputs": [],
"source": [
"import tensorflow as tf\n",
"print(tf.__version__)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "1Ua5FpcJNrIj"
},
"outputs": [],
"source": [
"import IPython.display as display\n",
"\n",
"import matplotlib.pyplot as plt\n",
"import matplotlib as mpl\n",
"mpl.rcParams['figure.figsize'] = (12,12)\n",
"mpl.rcParams['axes.grid'] = False\n",
"\n",
"import numpy as np\n",
"import time\n",
"import functools"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "1b988wrrQnVF"
},
"source": [
"Download the content and style images, and the pre-trained TensorFlow Lite models."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "16g57cIMQnen"
},
"outputs": [],
"source": [
"content_path = tf.keras.utils.get_file('belfry.jpg','https://storage.googleapis.com/khanhlvg-public.appspot.com/arbitrary-style-transfer/belfry-2611573_1280.jpg')\n",
"style_path = tf.keras.utils.get_file('style23.jpg','https://storage.googleapis.com/khanhlvg-public.appspot.com/arbitrary-style-transfer/style23.jpg')\n",
"\n",
"style_predict_path = tf.keras.utils.get_file('style_predict.tflite', 'https://tfhub.dev/google/lite-model/magenta/arbitrary-image-stylization-v1-256/int8/prediction/1?lite-format=tflite')\n",
"style_transform_path = tf.keras.utils.get_file('style_transform.tflite', 'https://tfhub.dev/google/lite-model/magenta/arbitrary-image-stylization-v1-256/int8/transfer/1?lite-format=tflite')"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "MQZXL7kON-gM"
},
"source": [
"## Pre-process the inputs\n",
"\n",
"* The content image and the style image must be RGB images with pixel values being float32 numbers between [0..1].\n",
"* The style image size must be (1, 256, 256, 3). We central crop the image and resize it.\n",
"* The content image must be (1, 384, 384, 3). We central crop the image and resize it."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Cg0Vi-rXRUFl"
},
"outputs": [],
"source": [
"# Function to load an image from a file, and add a batch dimension.\n",
"def load_img(path_to_img):\n",
" img = tf.io.read_file(path_to_img)\n",
" img = tf.io.decode_image(img, channels=3)\n",
" img = tf.image.convert_image_dtype(img, tf.float32)\n",
" img = img[tf.newaxis, :]\n",
"\n",
" return img\n",
"\n",
"# Function to pre-process by resizing an central cropping it.\n",
"def preprocess_image(image, target_dim):\n",
" # Resize the image so that the shorter dimension becomes 256px.\n",
" shape = tf.cast(tf.shape(image)[1:-1], tf.float32)\n",
" short_dim = min(shape)\n",
" scale = target_dim / short_dim\n",
" new_shape = tf.cast(shape * scale, tf.int32)\n",
" image = tf.image.resize(image, new_shape)\n",
"\n",
" # Central crop the image.\n",
" image = tf.image.resize_with_crop_or_pad(image, target_dim, target_dim)\n",
"\n",
" return image\n",
"\n",
"# Load the input images.\n",
"content_image = load_img(content_path)\n",
"style_image = load_img(style_path)\n",
"\n",
"# Preprocess the input images.\n",
"preprocessed_content_image = preprocess_image(content_image, 384)\n",
"preprocessed_style_image = preprocess_image(style_image, 256)\n",
"\n",
"print('Style Image Shape:', preprocessed_style_image.shape)\n",
"print('Content Image Shape:', preprocessed_content_image.shape)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "xE4Yt8nArTeR"
},
"source": [
"## Visualize the inputs"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ncPA4esJRcEu"
},
"outputs": [],
"source": [
"def imshow(image, title=None):\n",
" if len(image.shape) > 3:\n",
" image = tf.squeeze(image, axis=0)\n",
"\n",
" plt.imshow(image)\n",
" if title:\n",
" plt.title(title)\n",
"\n",
"plt.subplot(1, 2, 1)\n",
"imshow(preprocessed_content_image, 'Content Image')\n",
"\n",
"plt.subplot(1, 2, 2)\n",
"imshow(preprocessed_style_image, 'Style Image')"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "CJ7R-CHbjC3s"
},
"source": [
"## Run style transfer with TensorFlow Lite"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "euu00ldHjKwD"
},
"source": [
"### Style prediction"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "o3zd9cTFRiS_"
},
"outputs": [],
"source": [
"# Function to run style prediction on preprocessed style image.\n",
"def run_style_predict(preprocessed_style_image):\n",
" # Load the model.\n",
" interpreter = tf.lite.Interpreter(model_path=style_predict_path)\n",
"\n",
" # Set model input.\n",
" interpreter.allocate_tensors()\n",
" input_details = interpreter.get_input_details()\n",
" interpreter.set_tensor(input_details[0][\"index\"], preprocessed_style_image)\n",
"\n",
" # Calculate style bottleneck.\n",
" interpreter.invoke()\n",
" style_bottleneck = interpreter.tensor(\n",
" interpreter.get_output_details()[0][\"index\"]\n",
" )()\n",
"\n",
" return style_bottleneck\n",
"\n",
"# Calculate style bottleneck for the preprocessed style image.\n",
"style_bottleneck = run_style_predict(preprocessed_style_image)\n",
"print('Style Bottleneck Shape:', style_bottleneck.shape)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "00t8S2PekIyW"
},
"source": [
"### Style transform"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "cZp5bCj8SX1w"
},
"outputs": [],
"source": [
"# Run style transform on preprocessed style image\n",
"def run_style_transform(style_bottleneck, preprocessed_content_image):\n",
" # Load the model.\n",
" interpreter = tf.lite.Interpreter(model_path=style_transform_path)\n",
"\n",
" # Set model input.\n",
" input_details = interpreter.get_input_details()\n",
" interpreter.allocate_tensors()\n",
"\n",
" # Set model inputs.\n",
" interpreter.set_tensor(input_details[0][\"index\"], preprocessed_content_image)\n",
" interpreter.set_tensor(input_details[1][\"index\"], style_bottleneck)\n",
" interpreter.invoke()\n",
"\n",
" # Transform content image.\n",
" stylized_image = interpreter.tensor(\n",
" interpreter.get_output_details()[0][\"index\"]\n",
" )()\n",
"\n",
" return stylized_image\n",
"\n",
"# Stylize the content image using the style bottleneck.\n",
"stylized_image = run_style_transform(style_bottleneck, preprocessed_content_image)\n",
"\n",
"# Visualize the output.\n",
"imshow(stylized_image, 'Stylized Image')"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "vv_71Td-QtrW"
},
"source": [
"### Style blending\n",
"\n",
"We can blend the style of content image into the stylized output, which in turn making the output look more like the content image."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "eJcAURXQQtJ7"
},
"outputs": [],
"source": [
"# Calculate style bottleneck of the content image.\n",
"style_bottleneck_content = run_style_predict(\n",
" preprocess_image(content_image, 256)\n",
" )"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "4S3yg2MgkmRD"
},
"outputs": [],
"source": [
"# Define content blending ratio between [0..1].\n",
"# 0.0: 0% style extracts from content image.\n",
"# 1.0: 100% style extracted from content image.\n",
"content_blending_ratio = 0.5 #@param {type:\"slider\", min:0, max:1, step:0.01}\n",
"\n",
"# Blend the style bottleneck of style image and content image\n",
"style_bottleneck_blended = content_blending_ratio * style_bottleneck_content \\\n",
" + (1 - content_blending_ratio) * style_bottleneck\n",
"\n",
"# Stylize the content image using the style bottleneck.\n",
"stylized_image_blended = run_style_transform(style_bottleneck_blended,\n",
" preprocessed_content_image)\n",
"\n",
"# Visualize the output.\n",
"imshow(stylized_image_blended, 'Blended Stylized Image')"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "9k9jGIep8p1c"
},
"source": [
"## Performance Benchmarks\n",
"\n",
"Performance benchmark numbers are generated with the tool [described here](https://www.tensorflow.org/lite/performance/benchmarks).\n",
"<table ><thead><tr><th>Model name</th> <th>Model size</th> <th>Device </th> <th>NNAPI</th> <th>CPU</th> <th>GPU</th></tr> </thead> \n",
"<tr> <td rowspan = 3> <a href=\"https://tfhub.dev/google/lite-model/magenta/arbitrary-image-stylization-v1-256/int8/prediction/1?lite-format=tflite\">Style prediction model (int8)</a> </td> \n",
"<td rowspan = 3>2.8 Mb</td>\n",
"<td>Pixel 3 (Android 10) </td> <td>142ms</td><td>14ms*</td><td></td></tr>\n",
"<tr><td>Pixel 4 (Android 10) </td> <td>5.2ms</td><td>6.7ms*</td><td></td></tr>\n",
"<tr><td>iPhone XS (iOS 12.4.1) </td> <td></td><td>10.7ms**</td><td></td></tr>\n",
"<tr> <td rowspan = 3> <a href=\"https://tfhub.dev/google/lite-model/magenta/arbitrary-image-stylization-v1-256/int8/transfer/1?lite-format=tflite\">Style transform model (int8)</a> </td> \n",
"<td rowspan = 3>0.2 Mb</td>\n",
"<td>Pixel 3 (Android 10) </td> <td></td><td>540ms*</td><td></td></tr>\n",
"<tr><td>Pixel 4 (Android 10) </td> <td></td><td>405ms*</td><td></td></tr>\n",
"<tr><td>iPhone XS (iOS 12.4.1) </td> <td></td><td>251ms**</td><td></td></tr>\n",
"\n",
"<tr> <td rowspan = 2> <a href=\"https://tfhub.dev/google/lite-model/magenta/arbitrary-image-stylization-v1-256/fp16/prediction/1?lite-format=tflite\">Style prediction model (float16)</a> </td> \n",
"<td rowspan = 2>4.7 Mb</td>\n",
"<td>Pixel 3 (Android 10) </td> <td>86ms</td><td>28ms*</td><td>9.1ms</td></tr>\n",
"<tr><td>Pixel 4 (Android 10) </td><td>32ms</td><td>12ms*</td><td>10ms</td></tr>\n",
"\n",
"<tr> <td rowspan = 2> <a href=\"https://tfhub.dev/google/lite-model/magenta/arbitrary-image-stylization-v1-256/fp16/transfer/1?lite-format=tflite\">Style transfer model (float16)</a> </td> \n",
"<td rowspan = 2>0.4 Mb</td>\n",
"<td>Pixel 3 (Android 10) </td> <td>1095ms</td><td>545ms*</td><td>42ms</td></tr>\n",
"<tr><td>Pixel 4 (Android 10) </td><td>603ms</td><td>377ms*</td><td>42ms</td></tr>\n",
"\n",
"</table>\n",
"\n",
"*&ast; 4 threads used. <br/>*\n",
"*&ast;&ast; 2 threads on iPhone for the best performance.*\n"
]
}
],
"metadata": {
"colab": {
"collapsed_sections": [],
"name": "overview.ipynb",
"provenance": [],
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}