wehub/fchollet--deep-learning-with-python-notebooks
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 13:25:23 +08:00
commit 612b28061b
60 changed files with 49225 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
second_edition/README.md
+30
View File
@@ -0,0 +1,30 @@
# Second edition notebooks
These are the notebooks for the second edition of the book, originally published in 2021. These notebooks use `tf.keras` with TensorFlow 2.16.
## Table of contents
* [Chapter 2: The mathematical building blocks of neural networks](https://colab.research.google.com/github/fchollet/deep-learning-with-python-notebooks/blob/master/second_edition/chapter02_mathematical-building-blocks.ipynb)
* [Chapter 3: Introduction to Keras and TensorFlow](https://colab.research.google.com/github/fchollet/deep-learning-with-python-notebooks/blob/master/second_edition/chapter03_introduction-to-keras-and-tf.ipynb)
* [Chapter 4: Getting started with neural networks: classification and regression](https://colab.research.google.com/github/fchollet/deep-learning-with-python-notebooks/blob/master/second_edition/chapter04_getting-started-with-neural-networks.ipynb)
* [Chapter 5: Fundamentals of machine learning](https://colab.research.google.com/github/fchollet/deep-learning-with-python-notebooks/blob/master/second_edition/chapter05_fundamentals-of-ml.ipynb)
* [Chapter 7: Working with Keras: a deep dive](https://colab.research.google.com/github/fchollet/deep-learning-with-python-notebooks/blob/master/second_edition/chapter07_working-with-keras.ipynb)
* [Chapter 8: Introduction to deep learning for computer vision](https://colab.research.google.com/github/fchollet/deep-learning-with-python-notebooks/blob/master/second_edition/chapter08_intro-to-dl-for-computer-vision.ipynb)
* Chapter 9: Advanced deep learning for computer vision
- [Part 1: Image segmentation](https://colab.research.google.com/github/fchollet/deep-learning-with-python-notebooks/blob/master/second_edition/chapter09_part01_image-segmentation.ipynb)
- [Part 2: Modern convnet architecture patterns](https://colab.research.google.com/github/fchollet/deep-learning-with-python-notebooks/blob/master/second_edition/chapter09_part02_modern-convnet-architecture-patterns.ipynb)
- [Part 3: Interpreting what convnets learn](https://colab.research.google.com/github/fchollet/deep-learning-with-python-notebooks/blob/master/second_edition/chapter09_part03_interpreting-what-convnets-learn.ipynb)
* [Chapter 10: Deep learning for timeseries](https://colab.research.google.com/github/fchollet/deep-learning-with-python-notebooks/blob/master/second_edition/chapter10_dl-for-timeseries.ipynb)
* Chapter 11: Deep learning for text
- [Part 1: Introduction](https://colab.research.google.com/github/fchollet/deep-learning-with-python-notebooks/blob/master/second_edition/chapter11_part01_introduction.ipynb)
- [Part 2: Sequence models](https://colab.research.google.com/github/fchollet/deep-learning-with-python-notebooks/blob/master/second_edition/chapter11_part02_sequence-models.ipynb)
- [Part 3: Transformer](https://colab.research.google.com/github/fchollet/deep-learning-with-python-notebooks/blob/master/second_edition/chapter11_part03_transformer.ipynb)
- [Part 4: Sequence-to-sequence learning](https://colab.research.google.com/github/fchollet/deep-learning-with-python-notebooks/blob/master/second_edition/chapter11_part04_sequence-to-sequence-learning.ipynb)
* Chapter 12: Generative deep learning
- [Part 1: Text generation](https://colab.research.google.com/github/fchollet/deep-learning-with-python-notebooks/blob/master/second_edition/chapter12_part01_text-generation.ipynb)
- [Part 2: Deep Dream](https://colab.research.google.com/github/fchollet/deep-learning-with-python-notebooks/blob/master/second_edition/chapter12_part02_deep-dream.ipynb)
- [Part 3: Neural style transfer](https://colab.research.google.com/github/fchollet/deep-learning-with-python-notebooks/blob/master/second_edition/chapter12_part03_neural-style-transfer.ipynb)
- [Part 4: Variational autoencoders](https://colab.research.google.com/github/fchollet/deep-learning-with-python-notebooks/blob/master/second_edition/chapter12_part04_variational-autoencoders.ipynb)
- [Part 5: Generative adversarial networks](https://colab.research.google.com/github/fchollet/deep-learning-with-python-notebooks/blob/master/second_edition/chapter12_part05_gans.ipynb)
* [Chapter 13: Best practices for the real world](https://colab.research.google.com/github/fchollet/deep-learning-with-python-notebooks/blob/master/second_edition/chapter13_best-practices-for-the-real-world.ipynb)
* [Chapter 14: Conclusions](https://colab.research.google.com/github/fchollet/deep-learning-with-python-notebooks/blob/master/second_edition/chapter14_conclusions.ipynb)
second_edition/chapter02_mathematical-building-blocks.ipynb
+1469
View File
File diff suppressed because it is too large Load Diff
second_edition/chapter03_introduction-to-keras-and-tf.ipynb
+990
View File
@@ -0,0 +1,990 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"This is a companion notebook for the book [Deep Learning with Python, Second Edition](https://www.manning.com/books/deep-learning-with-python-second-edition?a_aid=keras&a_bid=76564dff). For readability, it only contains runnable code blocks and section titles, and omits everything else in the book: text paragraphs, figures, and pseudocode.\n\n**If you want to be able to follow what's going on, I recommend reading the notebook side by side with your copy of the book.**\n\nThis notebook was generated for TensorFlow 2.6."
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"# Introduction to Keras and TensorFlow"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## What's TensorFlow?"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## What's Keras?"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## Keras and TensorFlow: A brief history"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## Setting up a deep-learning workspace"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Jupyter notebooks: The preferred way to run deep-learning experiments"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Using Colaboratory"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### First steps with Colaboratory"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### Installing packages with pip"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### Using the GPU runtime"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## First steps with TensorFlow"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### Constant tensors and variables"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**All-ones or all-zeros tensors**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"import tensorflow as tf\n",
"x = tf.ones(shape=(2, 1))\n",
"print(x)"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"x = tf.zeros(shape=(2, 1))\n",
"print(x)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Random tensors**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"x = tf.random.normal(shape=(3, 1), mean=0., stddev=1.)\n",
"print(x)"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"x = tf.random.uniform(shape=(3, 1), minval=0., maxval=1.)\n",
"print(x)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**NumPy arrays are assignable**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"import numpy as np\n",
"x = np.ones(shape=(2, 2))\n",
"x[0, 0] = 0."
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Creating a TensorFlow variable**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"v = tf.Variable(initial_value=tf.random.normal(shape=(3, 1)))\n",
"print(v)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Assigning a value to a TensorFlow variable**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"v.assign(tf.ones((3, 1)))"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Assigning a value to a subset of a TensorFlow variable**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"v[0, 0].assign(3.)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Using `assign_add`**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"v.assign_add(tf.ones((3, 1)))"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### Tensor operations: Doing math in TensorFlow"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**A few basic math operations**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"a = tf.ones((2, 2))\n",
"b = tf.square(a)\n",
"c = tf.sqrt(a)\n",
"d = b + c\n",
"e = tf.matmul(a, b)\n",
"e *= d"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### A second look at the GradientTape API"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Using the `GradientTape`**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"input_var = tf.Variable(initial_value=3.)\n",
"with tf.GradientTape() as tape:\n",
" result = tf.square(input_var)\n",
"gradient = tape.gradient(result, input_var)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Using `GradientTape` with constant tensor inputs**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"input_const = tf.constant(3.)\n",
"with tf.GradientTape() as tape:\n",
" tape.watch(input_const)\n",
" result = tf.square(input_const)\n",
"gradient = tape.gradient(result, input_const)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Using nested gradient tapes to compute second-order gradients**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"time = tf.Variable(0.)\n",
"with tf.GradientTape() as outer_tape:\n",
" with tf.GradientTape() as inner_tape:\n",
" position = 4.9 * time ** 2\n",
" speed = inner_tape.gradient(position, time)\n",
"acceleration = outer_tape.gradient(speed, time)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### An end-to-end example: A linear classifier in pure TensorFlow"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Generating two classes of random points in a 2D plane**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"num_samples_per_class = 1000\n",
"negative_samples = np.random.multivariate_normal(\n",
" mean=[0, 3],\n",
" cov=[[1, 0.5],[0.5, 1]],\n",
" size=num_samples_per_class)\n",
"positive_samples = np.random.multivariate_normal(\n",
" mean=[3, 0],\n",
" cov=[[1, 0.5],[0.5, 1]],\n",
" size=num_samples_per_class)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Stacking the two classes into an array with shape (2000, 2)**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"inputs = np.vstack((negative_samples, positive_samples)).astype(np.float32)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Generating the corresponding targets (0 and 1)**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"targets = np.vstack((np.zeros((num_samples_per_class, 1), dtype=\"float32\"),\n",
" np.ones((num_samples_per_class, 1), dtype=\"float32\")))"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Plotting the two point classes**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"plt.scatter(inputs[:, 0], inputs[:, 1], c=targets[:, 0])\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Creating the linear classifier variables**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"input_dim = 2\n",
"output_dim = 1\n",
"W = tf.Variable(initial_value=tf.random.uniform(shape=(input_dim, output_dim)))\n",
"b = tf.Variable(initial_value=tf.zeros(shape=(output_dim,)))"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**The forward pass function**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"def model(inputs):\n",
" return tf.matmul(inputs, W) + b"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**The mean squared error loss function**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"def square_loss(targets, predictions):\n",
" per_sample_losses = tf.square(targets - predictions)\n",
" return tf.reduce_mean(per_sample_losses)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**The training step function**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"learning_rate = 0.1\n",
"\n",
"def training_step(inputs, targets):\n",
" with tf.GradientTape() as tape:\n",
" predictions = model(inputs)\n",
" loss = square_loss(targets, predictions)\n",
" grad_loss_wrt_W, grad_loss_wrt_b = tape.gradient(loss, [W, b])\n",
" W.assign_sub(grad_loss_wrt_W * learning_rate)\n",
" b.assign_sub(grad_loss_wrt_b * learning_rate)\n",
" return loss"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**The batch training loop**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"for step in range(40):\n",
" loss = training_step(inputs, targets)\n",
" print(f\"Loss at step {step}: {loss:.4f}\")"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"predictions = model(inputs)\n",
"plt.scatter(inputs[:, 0], inputs[:, 1], c=predictions[:, 0] > 0.5)\n",
"plt.show()"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"x = np.linspace(-1, 4, 100)\n",
"y = - W[0] / W[1] * x + (0.5 - b) / W[1]\n",
"plt.plot(x, y, \"-r\")\n",
"plt.scatter(inputs[:, 0], inputs[:, 1], c=predictions[:, 0] > 0.5)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## Anatomy of a neural network: Understanding core Keras APIs"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Layers: The building blocks of deep learning"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### The base Layer class in Keras"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**A `Dense` layer implemented as a `Layer` subclass**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"from tensorflow import keras\n",
"\n",
"class SimpleDense(keras.layers.Layer):\n",
"\n",
" def __init__(self, units, activation=None):\n",
" super().__init__()\n",
" self.units = units\n",
" self.activation = activation\n",
"\n",
" def build(self, input_shape):\n",
" input_dim = input_shape[-1]\n",
" self.W = self.add_weight(shape=(input_dim, self.units),\n",
" initializer=\"random_normal\")\n",
" self.b = self.add_weight(shape=(self.units,),\n",
" initializer=\"zeros\")\n",
"\n",
" def call(self, inputs):\n",
" y = tf.matmul(inputs, self.W) + self.b\n",
" if self.activation is not None:\n",
" y = self.activation(y)\n",
" return y"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"my_dense = SimpleDense(units=32, activation=tf.nn.relu)\n",
"input_tensor = tf.ones(shape=(2, 784))\n",
"output_tensor = my_dense(input_tensor)\n",
"print(output_tensor.shape)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### Automatic shape inference: Building layers on the fly"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"from tensorflow.keras import layers\n",
"layer = layers.Dense(32, activation=\"relu\")"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"from tensorflow.keras import models\n",
"from tensorflow.keras import layers\n",
"model = models.Sequential([\n",
" layers.Dense(32, activation=\"relu\"),\n",
" layers.Dense(32)\n",
"])"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"model = keras.Sequential([\n",
" SimpleDense(32, activation=\"relu\"),\n",
" SimpleDense(64, activation=\"relu\"),\n",
" SimpleDense(32, activation=\"relu\"),\n",
" SimpleDense(10, activation=\"softmax\")\n",
"])"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### From layers to models"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### The \"compile\" step: Configuring the learning process"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"model = keras.Sequential([keras.layers.Dense(1)])\n",
"model.compile(optimizer=\"rmsprop\",\n",
" loss=\"mean_squared_error\",\n",
" metrics=[\"accuracy\"])"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"model.compile(optimizer=keras.optimizers.RMSprop(),\n",
" loss=keras.losses.MeanSquaredError(),\n",
" metrics=[keras.metrics.BinaryAccuracy()])"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Picking a loss function"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Understanding the fit() method"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Calling `fit()` with NumPy data**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"history = model.fit(\n",
" inputs,\n",
" targets,\n",
" epochs=5,\n",
" batch_size=128\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"history.history"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Monitoring loss and metrics on validation data"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Using the `validation_data` argument**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"model = keras.Sequential([keras.layers.Dense(1)])\n",
"model.compile(optimizer=keras.optimizers.RMSprop(learning_rate=0.1),\n",
" loss=keras.losses.MeanSquaredError(),\n",
" metrics=[keras.metrics.BinaryAccuracy()])\n",
"\n",
"indices_permutation = np.random.permutation(len(inputs))\n",
"shuffled_inputs = inputs[indices_permutation]\n",
"shuffled_targets = targets[indices_permutation]\n",
"\n",
"num_validation_samples = int(0.3 * len(inputs))\n",
"val_inputs = shuffled_inputs[:num_validation_samples]\n",
"val_targets = shuffled_targets[:num_validation_samples]\n",
"training_inputs = shuffled_inputs[num_validation_samples:]\n",
"training_targets = shuffled_targets[num_validation_samples:]\n",
"model.fit(\n",
" training_inputs,\n",
" training_targets,\n",
" epochs=5,\n",
" batch_size=16,\n",
" validation_data=(val_inputs, val_targets)\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Inference: Using a model after training"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"predictions = model.predict(val_inputs, batch_size=128)\n",
"print(predictions[:10])"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## Summary"
]
}
],
"metadata": {
"colab": {
"collapsed_sections": [],
"name": "chapter03_introduction-to-keras-and-tf.i",
"private_outputs": false,
"provenance": [],
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.0"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
second_edition/chapter04_getting-started-with-neural-networks.ipynb
+1413
View File
File diff suppressed because it is too large Load Diff
second_edition/chapter05_fundamentals-of-ml.ipynb
+786
View File
@@ -0,0 +1,786 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"This is a companion notebook for the book [Deep Learning with Python, Second Edition](https://www.manning.com/books/deep-learning-with-python-second-edition?a_aid=keras&a_bid=76564dff). For readability, it only contains runnable code blocks and section titles, and omits everything else in the book: text paragraphs, figures, and pseudocode.\n\n**If you want to be able to follow what's going on, I recommend reading the notebook side by side with your copy of the book.**\n\nThis notebook was generated for TensorFlow 2.6."
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"# Fundamentals of machine learning"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## Generalization: The goal of machine learning"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Underfitting and overfitting"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### Noisy training data"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### Ambiguous features"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### Rare features and spurious correlations"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Adding white-noise channels or all-zeros channels to MNIST**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"from tensorflow.keras.datasets import mnist\n",
"import numpy as np\n",
"\n",
"(train_images, train_labels), _ = mnist.load_data()\n",
"train_images = train_images.reshape((60000, 28 * 28))\n",
"train_images = train_images.astype(\"float32\") / 255\n",
"\n",
"train_images_with_noise_channels = np.concatenate(\n",
" [train_images, np.random.random((len(train_images), 784))], axis=1)\n",
"\n",
"train_images_with_zeros_channels = np.concatenate(\n",
" [train_images, np.zeros((len(train_images), 784))], axis=1)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Training the same model on MNIST data with noise channels or all-zero channels**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"from tensorflow import keras\n",
"from tensorflow.keras import layers\n",
"\n",
"def get_model():\n",
" model = keras.Sequential([\n",
" layers.Dense(512, activation=\"relu\"),\n",
" layers.Dense(10, activation=\"softmax\")\n",
" ])\n",
" model.compile(optimizer=\"rmsprop\",\n",
" loss=\"sparse_categorical_crossentropy\",\n",
" metrics=[\"accuracy\"])\n",
" return model\n",
"\n",
"model = get_model()\n",
"history_noise = model.fit(\n",
" train_images_with_noise_channels, train_labels,\n",
" epochs=10,\n",
" batch_size=128,\n",
" validation_split=0.2)\n",
"\n",
"model = get_model()\n",
"history_zeros = model.fit(\n",
" train_images_with_zeros_channels, train_labels,\n",
" epochs=10,\n",
" batch_size=128,\n",
" validation_split=0.2)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Plotting a validation accuracy comparison**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"val_acc_noise = history_noise.history[\"val_accuracy\"]\n",
"val_acc_zeros = history_zeros.history[\"val_accuracy\"]\n",
"epochs = range(1, 11)\n",
"plt.plot(epochs, val_acc_noise, \"b-\",\n",
" label=\"Validation accuracy with noise channels\")\n",
"plt.plot(epochs, val_acc_zeros, \"b--\",\n",
" label=\"Validation accuracy with zeros channels\")\n",
"plt.title(\"Effect of noise channels on validation accuracy\")\n",
"plt.xlabel(\"Epochs\")\n",
"plt.ylabel(\"Accuracy\")\n",
"plt.legend()"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### The nature of generalization in deep learning"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Fitting a MNIST model with randomly shuffled labels**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"(train_images, train_labels), _ = mnist.load_data()\n",
"train_images = train_images.reshape((60000, 28 * 28))\n",
"train_images = train_images.astype(\"float32\") / 255\n",
"\n",
"random_train_labels = train_labels[:]\n",
"np.random.shuffle(random_train_labels)\n",
"\n",
"model = keras.Sequential([\n",
" layers.Dense(512, activation=\"relu\"),\n",
" layers.Dense(10, activation=\"softmax\")\n",
"])\n",
"model.compile(optimizer=\"rmsprop\",\n",
" loss=\"sparse_categorical_crossentropy\",\n",
" metrics=[\"accuracy\"])\n",
"model.fit(train_images, random_train_labels,\n",
" epochs=100,\n",
" batch_size=128,\n",
" validation_split=0.2)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### The manifold hypothesis"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### Interpolation as a source of generalization"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### Why deep learning works"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### Training data is paramount"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## Evaluating machine-learning models"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Training, validation, and test sets"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### Simple hold-out validation"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### K-fold validation"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### Iterated K-fold validation with shuffling"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Beating a common-sense baseline"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Things to keep in mind about model evaluation"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## Improving model fit"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Tuning key gradient descent parameters"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Training a MNIST model with an incorrectly high learning rate**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"(train_images, train_labels), _ = mnist.load_data()\n",
"train_images = train_images.reshape((60000, 28 * 28))\n",
"train_images = train_images.astype(\"float32\") / 255\n",
"\n",
"model = keras.Sequential([\n",
" layers.Dense(512, activation=\"relu\"),\n",
" layers.Dense(10, activation=\"softmax\")\n",
"])\n",
"model.compile(optimizer=keras.optimizers.RMSprop(1.),\n",
" loss=\"sparse_categorical_crossentropy\",\n",
" metrics=[\"accuracy\"])\n",
"model.fit(train_images, train_labels,\n",
" epochs=10,\n",
" batch_size=128,\n",
" validation_split=0.2)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**The same model with a more appropriate learning rate**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"model = keras.Sequential([\n",
" layers.Dense(512, activation=\"relu\"),\n",
" layers.Dense(10, activation=\"softmax\")\n",
"])\n",
"model.compile(optimizer=keras.optimizers.RMSprop(1e-2),\n",
" loss=\"sparse_categorical_crossentropy\",\n",
" metrics=[\"accuracy\"])\n",
"model.fit(train_images, train_labels,\n",
" epochs=10,\n",
" batch_size=128,\n",
" validation_split=0.2)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Leveraging better architecture priors"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Increasing model capacity"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**A simple logistic regression on MNIST**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"model = keras.Sequential([layers.Dense(10, activation=\"softmax\")])\n",
"model.compile(optimizer=\"rmsprop\",\n",
" loss=\"sparse_categorical_crossentropy\",\n",
" metrics=[\"accuracy\"])\n",
"history_small_model = model.fit(\n",
" train_images, train_labels,\n",
" epochs=20,\n",
" batch_size=128,\n",
" validation_split=0.2)"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"val_loss = history_small_model.history[\"val_loss\"]\n",
"epochs = range(1, 21)\n",
"plt.plot(epochs, val_loss, \"b--\",\n",
" label=\"Validation loss\")\n",
"plt.title(\"Effect of insufficient model capacity on validation loss\")\n",
"plt.xlabel(\"Epochs\")\n",
"plt.ylabel(\"Loss\")\n",
"plt.legend()"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"model = keras.Sequential([\n",
" layers.Dense(96, activation=\"relu\"),\n",
" layers.Dense(96, activation=\"relu\"),\n",
" layers.Dense(10, activation=\"softmax\"),\n",
"])\n",
"model.compile(optimizer=\"rmsprop\",\n",
" loss=\"sparse_categorical_crossentropy\",\n",
" metrics=[\"accuracy\"])\n",
"history_large_model = model.fit(\n",
" train_images, train_labels,\n",
" epochs=20,\n",
" batch_size=128,\n",
" validation_split=0.2)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## Improving generalization"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Dataset curation"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Feature engineering"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Using early stopping"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Regularizing your model"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### Reducing the network's size"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Original model**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"from tensorflow.keras.datasets import imdb\n",
"(train_data, train_labels), _ = imdb.load_data(num_words=10000)\n",
"\n",
"def vectorize_sequences(sequences, dimension=10000):\n",
" results = np.zeros((len(sequences), dimension))\n",
" for i, sequence in enumerate(sequences):\n",
" results[i, sequence] = 1.\n",
" return results\n",
"train_data = vectorize_sequences(train_data)\n",
"\n",
"model = keras.Sequential([\n",
" layers.Dense(16, activation=\"relu\"),\n",
" layers.Dense(16, activation=\"relu\"),\n",
" layers.Dense(1, activation=\"sigmoid\")\n",
"])\n",
"model.compile(optimizer=\"rmsprop\",\n",
" loss=\"binary_crossentropy\",\n",
" metrics=[\"accuracy\"])\n",
"history_original = model.fit(train_data, train_labels,\n",
" epochs=20, batch_size=512, validation_split=0.4)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Version of the model with lower capacity**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"model = keras.Sequential([\n",
" layers.Dense(4, activation=\"relu\"),\n",
" layers.Dense(4, activation=\"relu\"),\n",
" layers.Dense(1, activation=\"sigmoid\")\n",
"])\n",
"model.compile(optimizer=\"rmsprop\",\n",
" loss=\"binary_crossentropy\",\n",
" metrics=[\"accuracy\"])\n",
"history_smaller_model = model.fit(\n",
" train_data, train_labels,\n",
" epochs=20, batch_size=512, validation_split=0.4)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Version of the model with higher capacity**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"model = keras.Sequential([\n",
" layers.Dense(512, activation=\"relu\"),\n",
" layers.Dense(512, activation=\"relu\"),\n",
" layers.Dense(1, activation=\"sigmoid\")\n",
"])\n",
"model.compile(optimizer=\"rmsprop\",\n",
" loss=\"binary_crossentropy\",\n",
" metrics=[\"accuracy\"])\n",
"history_larger_model = model.fit(\n",
" train_data, train_labels,\n",
" epochs=20, batch_size=512, validation_split=0.4)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### Adding weight regularization"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Adding L2 weight regularization to the model**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"from tensorflow.keras import regularizers\n",
"model = keras.Sequential([\n",
" layers.Dense(16,\n",
" kernel_regularizer=regularizers.l2(0.002),\n",
" activation=\"relu\"),\n",
" layers.Dense(16,\n",
" kernel_regularizer=regularizers.l2(0.002),\n",
" activation=\"relu\"),\n",
" layers.Dense(1, activation=\"sigmoid\")\n",
"])\n",
"model.compile(optimizer=\"rmsprop\",\n",
" loss=\"binary_crossentropy\",\n",
" metrics=[\"accuracy\"])\n",
"history_l2_reg = model.fit(\n",
" train_data, train_labels,\n",
" epochs=20, batch_size=512, validation_split=0.4)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Different weight regularizers available in Keras**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"from tensorflow.keras import regularizers\n",
"regularizers.l1(0.001)\n",
"regularizers.l1_l2(l1=0.001, l2=0.001)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### Adding dropout"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Adding dropout to the IMDB model**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"model = keras.Sequential([\n",
" layers.Dense(16, activation=\"relu\"),\n",
" layers.Dropout(0.5),\n",
" layers.Dense(16, activation=\"relu\"),\n",
" layers.Dropout(0.5),\n",
" layers.Dense(1, activation=\"sigmoid\")\n",
"])\n",
"model.compile(optimizer=\"rmsprop\",\n",
" loss=\"binary_crossentropy\",\n",
" metrics=[\"accuracy\"])\n",
"history_dropout = model.fit(\n",
" train_data, train_labels,\n",
" epochs=20, batch_size=512, validation_split=0.4)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## Summary"
]
}
],
"metadata": {
"colab": {
"collapsed_sections": [],
"name": "chapter05_fundamentals-of-ml.i",
"private_outputs": false,
"provenance": [],
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.0"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
second_edition/chapter07_working-with-keras.ipynb
+1439
View File
File diff suppressed because it is too large Load Diff
second_edition/chapter08_intro-to-dl-for-computer-vision.ipynb
+1224
View File
File diff suppressed because it is too large Load Diff
second_edition/chapter09_part01_image-segmentation.ipynb
+282
View File
@@ -0,0 +1,282 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"This is a companion notebook for the book [Deep Learning with Python, Second Edition](https://www.manning.com/books/deep-learning-with-python-second-edition?a_aid=keras&a_bid=76564dff). For readability, it only contains runnable code blocks and section titles, and omits everything else in the book: text paragraphs, figures, and pseudocode.\n\n**If you want to be able to follow what's going on, I recommend reading the notebook side by side with your copy of the book.**\n\nThis notebook was generated for TensorFlow 2.6."
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"# Advanced deep learning for computer vision"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## Three essential computer vision tasks"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## An image segmentation example"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"!wget http://www.robots.ox.ac.uk/~vgg/data/pets/data/images.tar.gz\n",
"!wget http://www.robots.ox.ac.uk/~vgg/data/pets/data/annotations.tar.gz\n",
"!tar -xf images.tar.gz\n",
"!tar -xf annotations.tar.gz"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"import os\n",
"\n",
"input_dir = \"images/\"\n",
"target_dir = \"annotations/trimaps/\"\n",
"\n",
"input_img_paths = sorted(\n",
" [os.path.join(input_dir, fname)\n",
" for fname in os.listdir(input_dir)\n",
" if fname.endswith(\".jpg\")])\n",
"target_paths = sorted(\n",
" [os.path.join(target_dir, fname)\n",
" for fname in os.listdir(target_dir)\n",
" if fname.endswith(\".png\") and not fname.startswith(\".\")])"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"from tensorflow.keras.utils import load_img, img_to_array\n",
"\n",
"plt.axis(\"off\")\n",
"plt.imshow(load_img(input_img_paths[9]))"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"def display_target(target_array):\n",
" normalized_array = (target_array.astype(\"uint8\") - 1) * 127\n",
" plt.axis(\"off\")\n",
" plt.imshow(normalized_array[:, :, 0])\n",
"\n",
"img = img_to_array(load_img(target_paths[9], color_mode=\"grayscale\"))\n",
"display_target(img)"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"import numpy as np\n",
"import random\n",
"\n",
"img_size = (200, 200)\n",
"num_imgs = len(input_img_paths)\n",
"\n",
"random.Random(1337).shuffle(input_img_paths)\n",
"random.Random(1337).shuffle(target_paths)\n",
"\n",
"def path_to_input_image(path):\n",
" return img_to_array(load_img(path, target_size=img_size))\n",
"\n",
"def path_to_target(path):\n",
" img = img_to_array(\n",
" load_img(path, target_size=img_size, color_mode=\"grayscale\"))\n",
" img = img.astype(\"uint8\") - 1\n",
" return img\n",
"\n",
"input_imgs = np.zeros((num_imgs,) + img_size + (3,), dtype=\"float32\")\n",
"targets = np.zeros((num_imgs,) + img_size + (1,), dtype=\"uint8\")\n",
"for i in range(num_imgs):\n",
" input_imgs[i] = path_to_input_image(input_img_paths[i])\n",
" targets[i] = path_to_target(target_paths[i])\n",
"\n",
"num_val_samples = 1000\n",
"train_input_imgs = input_imgs[:-num_val_samples]\n",
"train_targets = targets[:-num_val_samples]\n",
"val_input_imgs = input_imgs[-num_val_samples:]\n",
"val_targets = targets[-num_val_samples:]"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"from tensorflow import keras\n",
"from tensorflow.keras import layers\n",
"\n",
"def get_model(img_size, num_classes):\n",
" inputs = keras.Input(shape=img_size + (3,))\n",
" x = layers.Rescaling(1./255)(inputs)\n",
"\n",
" x = layers.Conv2D(64, 3, strides=2, activation=\"relu\", padding=\"same\")(x)\n",
" x = layers.Conv2D(64, 3, activation=\"relu\", padding=\"same\")(x)\n",
" x = layers.Conv2D(128, 3, strides=2, activation=\"relu\", padding=\"same\")(x)\n",
" x = layers.Conv2D(128, 3, activation=\"relu\", padding=\"same\")(x)\n",
" x = layers.Conv2D(256, 3, strides=2, padding=\"same\", activation=\"relu\")(x)\n",
" x = layers.Conv2D(256, 3, activation=\"relu\", padding=\"same\")(x)\n",
"\n",
" x = layers.Conv2DTranspose(256, 3, activation=\"relu\", padding=\"same\")(x)\n",
" x = layers.Conv2DTranspose(256, 3, activation=\"relu\", padding=\"same\", strides=2)(x)\n",
" x = layers.Conv2DTranspose(128, 3, activation=\"relu\", padding=\"same\")(x)\n",
" x = layers.Conv2DTranspose(128, 3, activation=\"relu\", padding=\"same\", strides=2)(x)\n",
" x = layers.Conv2DTranspose(64, 3, activation=\"relu\", padding=\"same\")(x)\n",
" x = layers.Conv2DTranspose(64, 3, activation=\"relu\", padding=\"same\", strides=2)(x)\n",
"\n",
" outputs = layers.Conv2D(num_classes, 3, activation=\"softmax\", padding=\"same\")(x)\n",
"\n",
" model = keras.Model(inputs, outputs)\n",
" return model\n",
"\n",
"model = get_model(img_size=img_size, num_classes=3)\n",
"model.summary()"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"model.compile(optimizer=\"rmsprop\", loss=\"sparse_categorical_crossentropy\")\n",
"\n",
"callbacks = [\n",
" keras.callbacks.ModelCheckpoint(\"oxford_segmentation.keras\",\n",
" save_best_only=True)\n",
"]\n",
"\n",
"history = model.fit(train_input_imgs, train_targets,\n",
" epochs=50,\n",
" callbacks=callbacks,\n",
" batch_size=64,\n",
" validation_data=(val_input_imgs, val_targets))"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"epochs = range(1, len(history.history[\"loss\"]) + 1)\n",
"loss = history.history[\"loss\"]\n",
"val_loss = history.history[\"val_loss\"]\n",
"plt.figure()\n",
"plt.plot(epochs, loss, \"bo\", label=\"Training loss\")\n",
"plt.plot(epochs, val_loss, \"b\", label=\"Validation loss\")\n",
"plt.title(\"Training and validation loss\")\n",
"plt.legend()"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"from tensorflow.keras.utils import array_to_img\n",
"\n",
"model = keras.models.load_model(\"oxford_segmentation.keras\")\n",
"\n",
"i = 4\n",
"test_image = val_input_imgs[i]\n",
"plt.axis(\"off\")\n",
"plt.imshow(array_to_img(test_image))\n",
"\n",
"mask = model.predict(np.expand_dims(test_image, 0))[0]\n",
"\n",
"def display_mask(pred):\n",
" mask = np.argmax(pred, axis=-1)\n",
" mask *= 127\n",
" plt.axis(\"off\")\n",
" plt.imshow(mask)\n",
"\n",
"display_mask(mask)"
]
}
],
"metadata": {
"colab": {
"collapsed_sections": [],
"name": "chapter09_part01_image-segmentation.i",
"private_outputs": false,
"provenance": [],
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.0"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
second_edition/chapter09_part02_modern-convnet-architecture-patterns.ipynb
+321
View File
@@ -0,0 +1,321 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"This is a companion notebook for the book [Deep Learning with Python, Second Edition](https://www.manning.com/books/deep-learning-with-python-second-edition?a_aid=keras&a_bid=76564dff). For readability, it only contains runnable code blocks and section titles, and omits everything else in the book: text paragraphs, figures, and pseudocode.\n\n**If you want to be able to follow what's going on, I recommend reading the notebook side by side with your copy of the book.**\n\nThis notebook was generated for TensorFlow 2.6."
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## Modern convnet architecture patterns"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Modularity, hierarchy, and reuse"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Residual connections"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Residual block where the number of filters changes**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"from tensorflow import keras\n",
"from tensorflow.keras import layers\n",
"\n",
"inputs = keras.Input(shape=(32, 32, 3))\n",
"x = layers.Conv2D(32, 3, activation=\"relu\")(inputs)\n",
"residual = x\n",
"x = layers.Conv2D(64, 3, activation=\"relu\", padding=\"same\")(x)\n",
"residual = layers.Conv2D(64, 1)(residual)\n",
"x = layers.add([x, residual])"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Case where target block includes a max pooling layer**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"inputs = keras.Input(shape=(32, 32, 3))\n",
"x = layers.Conv2D(32, 3, activation=\"relu\")(inputs)\n",
"residual = x\n",
"x = layers.Conv2D(64, 3, activation=\"relu\", padding=\"same\")(x)\n",
"x = layers.MaxPooling2D(2, padding=\"same\")(x)\n",
"residual = layers.Conv2D(64, 1, strides=2)(residual)\n",
"x = layers.add([x, residual])"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"inputs = keras.Input(shape=(32, 32, 3))\n",
"x = layers.Rescaling(1./255)(inputs)\n",
"\n",
"def residual_block(x, filters, pooling=False):\n",
" residual = x\n",
" x = layers.Conv2D(filters, 3, activation=\"relu\", padding=\"same\")(x)\n",
" x = layers.Conv2D(filters, 3, activation=\"relu\", padding=\"same\")(x)\n",
" if pooling:\n",
" x = layers.MaxPooling2D(2, padding=\"same\")(x)\n",
" residual = layers.Conv2D(filters, 1, strides=2)(residual)\n",
" elif filters != residual.shape[-1]:\n",
" residual = layers.Conv2D(filters, 1)(residual)\n",
" x = layers.add([x, residual])\n",
" return x\n",
"\n",
"x = residual_block(x, filters=32, pooling=True)\n",
"x = residual_block(x, filters=64, pooling=True)\n",
"x = residual_block(x, filters=128, pooling=False)\n",
"\n",
"x = layers.GlobalAveragePooling2D()(x)\n",
"outputs = layers.Dense(1, activation=\"sigmoid\")(x)\n",
"model = keras.Model(inputs=inputs, outputs=outputs)\n",
"model.summary()"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Batch normalization"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Depthwise separable convolutions"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Putting it together: A mini Xception-like model"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"from google.colab import files\n",
"files.upload()"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"!mkdir ~/.kaggle\n",
"!cp kaggle.json ~/.kaggle/\n",
"!chmod 600 ~/.kaggle/kaggle.json\n",
"!kaggle competitions download -c dogs-vs-cats\n",
"!unzip -qq train.zip"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"import os, shutil, pathlib\n",
"from tensorflow.keras.utils import image_dataset_from_directory\n",
"\n",
"original_dir = pathlib.Path(\"train\")\n",
"new_base_dir = pathlib.Path(\"cats_vs_dogs_small\")\n",
"\n",
"def make_subset(subset_name, start_index, end_index):\n",
" for category in (\"cat\", \"dog\"):\n",
" dir = new_base_dir / subset_name / category\n",
" os.makedirs(dir)\n",
" fnames = [f\"{category}.{i}.jpg\" for i in range(start_index, end_index)]\n",
" for fname in fnames:\n",
" shutil.copyfile(src=original_dir / fname,\n",
" dst=dir / fname)\n",
"\n",
"make_subset(\"train\", start_index=0, end_index=1000)\n",
"make_subset(\"validation\", start_index=1000, end_index=1500)\n",
"make_subset(\"test\", start_index=1500, end_index=2500)\n",
"\n",
"train_dataset = image_dataset_from_directory(\n",
" new_base_dir / \"train\",\n",
" image_size=(180, 180),\n",
" batch_size=32)\n",
"validation_dataset = image_dataset_from_directory(\n",
" new_base_dir / \"validation\",\n",
" image_size=(180, 180),\n",
" batch_size=32)\n",
"test_dataset = image_dataset_from_directory(\n",
" new_base_dir / \"test\",\n",
" image_size=(180, 180),\n",
" batch_size=32)"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"data_augmentation = keras.Sequential(\n",
" [\n",
" layers.RandomFlip(\"horizontal\"),\n",
" layers.RandomRotation(0.1),\n",
" layers.RandomZoom(0.2),\n",
" ]\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"inputs = keras.Input(shape=(180, 180, 3))\n",
"x = data_augmentation(inputs)\n",
"\n",
"x = layers.Rescaling(1./255)(x)\n",
"x = layers.Conv2D(filters=32, kernel_size=5, use_bias=False)(x)\n",
"\n",
"for size in [32, 64, 128, 256, 512]:\n",
" residual = x\n",
"\n",
" x = layers.BatchNormalization()(x)\n",
" x = layers.Activation(\"relu\")(x)\n",
" x = layers.SeparableConv2D(size, 3, padding=\"same\", use_bias=False)(x)\n",
"\n",
" x = layers.BatchNormalization()(x)\n",
" x = layers.Activation(\"relu\")(x)\n",
" x = layers.SeparableConv2D(size, 3, padding=\"same\", use_bias=False)(x)\n",
"\n",
" x = layers.MaxPooling2D(3, strides=2, padding=\"same\")(x)\n",
"\n",
" residual = layers.Conv2D(\n",
" size, 1, strides=2, padding=\"same\", use_bias=False)(residual)\n",
" x = layers.add([x, residual])\n",
"\n",
"x = layers.GlobalAveragePooling2D()(x)\n",
"x = layers.Dropout(0.5)(x)\n",
"outputs = layers.Dense(1, activation=\"sigmoid\")(x)\n",
"model = keras.Model(inputs=inputs, outputs=outputs)"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"model.compile(loss=\"binary_crossentropy\",\n",
" optimizer=\"rmsprop\",\n",
" metrics=[\"accuracy\"])\n",
"history = model.fit(\n",
" train_dataset,\n",
" epochs=100,\n",
" validation_data=validation_dataset)"
]
}
],
"metadata": {
"colab": {
"collapsed_sections": [],
"name": "chapter09_part02_modern-convnet-architecture-patterns.i",
"private_outputs": false,
"provenance": [],
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.0"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
second_edition/chapter09_part03_interpreting-what-convnets-learn.ipynb
+785
View File
@@ -0,0 +1,785 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"This is a companion notebook for the book [Deep Learning with Python, Second Edition](https://www.manning.com/books/deep-learning-with-python-second-edition?a_aid=keras&a_bid=76564dff). For readability, it only contains runnable code blocks and section titles, and omits everything else in the book: text paragraphs, figures, and pseudocode.\n\n**If you want to be able to follow what's going on, I recommend reading the notebook side by side with your copy of the book.**\n\nThis notebook was generated for TensorFlow 2.6."
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## Interpreting what convnets learn"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Visualizing intermediate activations"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"# You can use this to load the file \"convnet_from_scratch_with_augmentation.keras\"\n",
"# you obtained in the last chapter.\n",
"from google.colab import files\n",
"files.upload()"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"from tensorflow import keras\n",
"model = keras.models.load_model(\"convnet_from_scratch_with_augmentation.keras\")\n",
"model.summary()"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Preprocessing a single image**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"from tensorflow import keras\n",
"import numpy as np\n",
"\n",
"img_path = keras.utils.get_file(\n",
" fname=\"cat.jpg\",\n",
" origin=\"https://img-datasets.s3.amazonaws.com/cat.jpg\")\n",
"\n",
"def get_img_array(img_path, target_size):\n",
" img = keras.utils.load_img(\n",
" img_path, target_size=target_size)\n",
" array = keras.utils.img_to_array(img)\n",
" array = np.expand_dims(array, axis=0)\n",
" return array\n",
"\n",
"img_tensor = get_img_array(img_path, target_size=(180, 180))"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Displaying the test picture**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"plt.axis(\"off\")\n",
"plt.imshow(img_tensor[0].astype(\"uint8\"))\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Instantiating a model that returns layer activations**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"from tensorflow.keras import layers\n",
"\n",
"layer_outputs = []\n",
"layer_names = []\n",
"for layer in model.layers:\n",
" if isinstance(layer, (layers.Conv2D, layers.MaxPooling2D)):\n",
" layer_outputs.append(layer.output)\n",
" layer_names.append(layer.name)\n",
"activation_model = keras.Model(inputs=model.input, outputs=layer_outputs)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Using the model to compute layer activations**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"activations = activation_model.predict(img_tensor)"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"first_layer_activation = activations[0]\n",
"print(first_layer_activation.shape)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Visualizing the fifth channel**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"plt.matshow(first_layer_activation[0, :, :, 5], cmap=\"viridis\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Visualizing every channel in every intermediate activation**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"images_per_row = 16\n",
"for layer_name, layer_activation in zip(layer_names, activations):\n",
" n_features = layer_activation.shape[-1]\n",
" size = layer_activation.shape[1]\n",
" n_cols = n_features // images_per_row\n",
" display_grid = np.zeros(((size + 1) * n_cols - 1,\n",
" images_per_row * (size + 1) - 1))\n",
" for col in range(n_cols):\n",
" for row in range(images_per_row):\n",
" channel_index = col * images_per_row + row\n",
" channel_image = layer_activation[0, :, :, channel_index].copy()\n",
" if channel_image.sum() != 0:\n",
" channel_image -= channel_image.mean()\n",
" channel_image /= channel_image.std()\n",
" channel_image *= 64\n",
" channel_image += 128\n",
" channel_image = np.clip(channel_image, 0, 255).astype(\"uint8\")\n",
" display_grid[\n",
" col * (size + 1): (col + 1) * size + col,\n",
" row * (size + 1) : (row + 1) * size + row] = channel_image\n",
" scale = 1. / size\n",
" plt.figure(figsize=(scale * display_grid.shape[1],\n",
" scale * display_grid.shape[0]))\n",
" plt.title(layer_name)\n",
" plt.grid(False)\n",
" plt.axis(\"off\")\n",
" plt.imshow(display_grid, aspect=\"auto\", cmap=\"viridis\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Visualizing convnet filters"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Instantiating the Xception convolutional base**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"model = keras.applications.xception.Xception(\n",
" weights=\"imagenet\",\n",
" include_top=False)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Printing the names of all convolutional layers in Xception**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"for layer in model.layers:\n",
" if isinstance(layer, (keras.layers.Conv2D, keras.layers.SeparableConv2D)):\n",
" print(layer.name)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Creating a feature extractor model**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"layer_name = \"block3_sepconv1\"\n",
"layer = model.get_layer(name=layer_name)\n",
"feature_extractor = keras.Model(inputs=model.input, outputs=layer.output)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Using the feature extractor**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"activation = feature_extractor(\n",
" keras.applications.xception.preprocess_input(img_tensor)\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"import tensorflow as tf\n",
"\n",
"def compute_loss(image, filter_index):\n",
" activation = feature_extractor(image)\n",
" filter_activation = activation[:, 2:-2, 2:-2, filter_index]\n",
" return tf.reduce_mean(filter_activation)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Loss maximization via stochastic gradient ascent**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"@tf.function\n",
"def gradient_ascent_step(image, filter_index, learning_rate):\n",
" with tf.GradientTape() as tape:\n",
" tape.watch(image)\n",
" loss = compute_loss(image, filter_index)\n",
" grads = tape.gradient(loss, image)\n",
" grads = tf.math.l2_normalize(grads)\n",
" image += learning_rate * grads\n",
" return image"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Function to generate filter visualizations**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"img_width = 200\n",
"img_height = 200\n",
"\n",
"def generate_filter_pattern(filter_index):\n",
" iterations = 30\n",
" learning_rate = 10.\n",
" image = tf.random.uniform(\n",
" minval=0.4,\n",
" maxval=0.6,\n",
" shape=(1, img_width, img_height, 3))\n",
" for i in range(iterations):\n",
" image = gradient_ascent_step(image, filter_index, learning_rate)\n",
" return image[0].numpy()"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Utility function to convert a tensor into a valid image**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"def deprocess_image(image):\n",
" image -= image.mean()\n",
" image /= image.std()\n",
" image *= 64\n",
" image += 128\n",
" image = np.clip(image, 0, 255).astype(\"uint8\")\n",
" image = image[25:-25, 25:-25, :]\n",
" return image"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"plt.axis(\"off\")\n",
"plt.imshow(deprocess_image(generate_filter_pattern(filter_index=2)))"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Generating a grid of all filter response patterns in a layer**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"all_images = []\n",
"for filter_index in range(64):\n",
" print(f\"Processing filter {filter_index}\")\n",
" image = deprocess_image(\n",
" generate_filter_pattern(filter_index)\n",
" )\n",
" all_images.append(image)\n",
"\n",
"margin = 5\n",
"n = 8\n",
"cropped_width = img_width - 25 * 2\n",
"cropped_height = img_height - 25 * 2\n",
"width = n * cropped_width + (n - 1) * margin\n",
"height = n * cropped_height + (n - 1) * margin\n",
"stitched_filters = np.zeros((width, height, 3))\n",
"\n",
"for i in range(n):\n",
" for j in range(n):\n",
" image = all_images[i * n + j]\n",
" stitched_filters[\n",
" (cropped_width + margin) * i : (cropped_width + margin) * i + cropped_width,\n",
" (cropped_height + margin) * j : (cropped_height + margin) * j\n",
" + cropped_height,\n",
" :,\n",
" ] = image\n",
"\n",
"keras.utils.save_img(\n",
" f\"filters_for_layer_{layer_name}.png\", stitched_filters)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Visualizing heatmaps of class activation"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Loading the Xception network with pretrained weights**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"model = keras.applications.xception.Xception(weights=\"imagenet\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Preprocessing an input image for Xception**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"img_path = keras.utils.get_file(\n",
" fname=\"elephant.jpg\",\n",
" origin=\"https://img-datasets.s3.amazonaws.com/elephant.jpg\")\n",
"\n",
"def get_img_array(img_path, target_size):\n",
" img = keras.utils.load_img(img_path, target_size=target_size)\n",
" array = keras.utils.img_to_array(img)\n",
" array = np.expand_dims(array, axis=0)\n",
" array = keras.applications.xception.preprocess_input(array)\n",
" return array\n",
"\n",
"img_array = get_img_array(img_path, target_size=(299, 299))"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"preds = model.predict(img_array)\n",
"print(keras.applications.xception.decode_predictions(preds, top=3)[0])"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"np.argmax(preds[0])"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Setting up a model that returns the last convolutional output**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"last_conv_layer_name = \"block14_sepconv2_act\"\n",
"classifier_layer_names = [\n",
" \"avg_pool\",\n",
" \"predictions\",\n",
"]\n",
"last_conv_layer = model.get_layer(last_conv_layer_name)\n",
"last_conv_layer_model = keras.Model(model.inputs, last_conv_layer.output)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Reapplying the classifier on top of the last convolutional output**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"classifier_input = keras.Input(shape=last_conv_layer.output.shape[1:])\n",
"x = classifier_input\n",
"for layer_name in classifier_layer_names:\n",
" x = model.get_layer(layer_name)(x)\n",
"classifier_model = keras.Model(classifier_input, x)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Retrieving the gradients of the top predicted class**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"import tensorflow as tf\n",
"\n",
"with tf.GradientTape() as tape:\n",
" last_conv_layer_output = last_conv_layer_model(img_array)\n",
" tape.watch(last_conv_layer_output)\n",
" preds = classifier_model(last_conv_layer_output)\n",
" top_pred_index = tf.argmax(preds[0])\n",
" top_class_channel = preds[:, top_pred_index]\n",
"\n",
"grads = tape.gradient(top_class_channel, last_conv_layer_output)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Gradient pooling and channel-importance weighting**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"pooled_grads = tf.reduce_mean(grads, axis=(0, 1, 2)).numpy()\n",
"last_conv_layer_output = last_conv_layer_output.numpy()[0]\n",
"for i in range(pooled_grads.shape[-1]):\n",
" last_conv_layer_output[:, :, i] *= pooled_grads[i]\n",
"heatmap = np.mean(last_conv_layer_output, axis=-1)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Heatmap post-processing**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"heatmap = np.maximum(heatmap, 0)\n",
"heatmap /= np.max(heatmap)\n",
"plt.matshow(heatmap)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Superimposing the heatmap on the original picture**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"import matplotlib.cm as cm\n",
"\n",
"img = keras.utils.load_img(img_path)\n",
"img = keras.utils.img_to_array(img)\n",
"\n",
"heatmap = np.uint8(255 * heatmap)\n",
"\n",
"jet = cm.get_cmap(\"jet\")\n",
"jet_colors = jet(np.arange(256))[:, :3]\n",
"jet_heatmap = jet_colors[heatmap]\n",
"\n",
"jet_heatmap = keras.utils.array_to_img(jet_heatmap)\n",
"jet_heatmap = jet_heatmap.resize((img.shape[1], img.shape[0]))\n",
"jet_heatmap = keras.utils.img_to_array(jet_heatmap)\n",
"\n",
"superimposed_img = jet_heatmap * 0.4 + img\n",
"superimposed_img = keras.utils.array_to_img(superimposed_img)\n",
"\n",
"save_path = \"elephant_cam.jpg\"\n",
"superimposed_img.save(save_path)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## Summary"
]
}
],
"metadata": {
"colab": {
"collapsed_sections": [],
"name": "chapter09_part03_interpreting-what-convnets-learn.i",
"private_outputs": false,
"provenance": [],
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.0"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
second_edition/chapter10_dl-for-timeseries.ipynb
+845
View File
@@ -0,0 +1,845 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"This is a companion notebook for the book [Deep Learning with Python, Second Edition](https://www.manning.com/books/deep-learning-with-python-second-edition?a_aid=keras&a_bid=76564dff). For readability, it only contains runnable code blocks and section titles, and omits everything else in the book: text paragraphs, figures, and pseudocode.\n\n**If you want to be able to follow what's going on, I recommend reading the notebook side by side with your copy of the book.**\n\nThis notebook was generated for TensorFlow 2.6."
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"# Deep learning for timeseries"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## Different kinds of timeseries tasks"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## A temperature-forecasting example"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"!wget https://s3.amazonaws.com/keras-datasets/jena_climate_2009_2016.csv.zip\n",
"!unzip jena_climate_2009_2016.csv.zip"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Inspecting the data of the Jena weather dataset**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"import os\n",
"fname = os.path.join(\"jena_climate_2009_2016.csv\")\n",
"\n",
"with open(fname) as f:\n",
" data = f.read()\n",
"\n",
"lines = data.split(\"\\n\")\n",
"header = lines[0].split(\",\")\n",
"lines = lines[1:]\n",
"print(header)\n",
"print(len(lines))"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Parsing the data**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"import numpy as np\n",
"temperature = np.zeros((len(lines),))\n",
"raw_data = np.zeros((len(lines), len(header) - 1))\n",
"for i, line in enumerate(lines):\n",
" values = [float(x) for x in line.split(\",\")[1:]]\n",
" temperature[i] = values[1]\n",
" raw_data[i, :] = values[:]"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Plotting the temperature timeseries**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"from matplotlib import pyplot as plt\n",
"plt.plot(range(len(temperature)), temperature)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Plotting the first 10 days of the temperature timeseries**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"plt.plot(range(1440), temperature[:1440])"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Computing the number of samples we'll use for each data split**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"num_train_samples = int(0.5 * len(raw_data))\n",
"num_val_samples = int(0.25 * len(raw_data))\n",
"num_test_samples = len(raw_data) - num_train_samples - num_val_samples\n",
"print(\"num_train_samples:\", num_train_samples)\n",
"print(\"num_val_samples:\", num_val_samples)\n",
"print(\"num_test_samples:\", num_test_samples)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Preparing the data"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Normalizing the data**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"mean = raw_data[:num_train_samples].mean(axis=0)\n",
"raw_data -= mean\n",
"std = raw_data[:num_train_samples].std(axis=0)\n",
"raw_data /= std"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"import numpy as np\n",
"from tensorflow import keras\n",
"int_sequence = np.arange(10)\n",
"dummy_dataset = keras.utils.timeseries_dataset_from_array(\n",
" data=int_sequence[:-3],\n",
" targets=int_sequence[3:],\n",
" sequence_length=3,\n",
" batch_size=2,\n",
")\n",
"\n",
"for inputs, targets in dummy_dataset:\n",
" for i in range(inputs.shape[0]):\n",
" print([int(x) for x in inputs[i]], int(targets[i]))"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Instantiating datasets for training, validation, and testing**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"sampling_rate = 6\n",
"sequence_length = 120\n",
"delay = sampling_rate * (sequence_length + 24 - 1)\n",
"batch_size = 256\n",
"\n",
"train_dataset = keras.utils.timeseries_dataset_from_array(\n",
" raw_data[:-delay],\n",
" targets=temperature[delay:],\n",
" sampling_rate=sampling_rate,\n",
" sequence_length=sequence_length,\n",
" shuffle=True,\n",
" batch_size=batch_size,\n",
" start_index=0,\n",
" end_index=num_train_samples)\n",
"\n",
"val_dataset = keras.utils.timeseries_dataset_from_array(\n",
" raw_data[:-delay],\n",
" targets=temperature[delay:],\n",
" sampling_rate=sampling_rate,\n",
" sequence_length=sequence_length,\n",
" shuffle=True,\n",
" batch_size=batch_size,\n",
" start_index=num_train_samples,\n",
" end_index=num_train_samples + num_val_samples)\n",
"\n",
"test_dataset = keras.utils.timeseries_dataset_from_array(\n",
" raw_data[:-delay],\n",
" targets=temperature[delay:],\n",
" sampling_rate=sampling_rate,\n",
" sequence_length=sequence_length,\n",
" shuffle=True,\n",
" batch_size=batch_size,\n",
" start_index=num_train_samples + num_val_samples)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Inspecting the output of one of our datasets**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"for samples, targets in train_dataset:\n",
" print(\"samples shape:\", samples.shape)\n",
" print(\"targets shape:\", targets.shape)\n",
" break"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### A common-sense, non-machine-learning baseline"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Computing the common-sense baseline MAE**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"def evaluate_naive_method(dataset):\n",
" total_abs_err = 0.\n",
" samples_seen = 0\n",
" for samples, targets in dataset:\n",
" preds = samples[:, -1, 1] * std[1] + mean[1]\n",
" total_abs_err += np.sum(np.abs(preds - targets))\n",
" samples_seen += samples.shape[0]\n",
" return total_abs_err / samples_seen\n",
"\n",
"print(f\"Validation MAE: {evaluate_naive_method(val_dataset):.2f}\")\n",
"print(f\"Test MAE: {evaluate_naive_method(test_dataset):.2f}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Let's try a basic machine-learning model"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Training and evaluating a densely connected model**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"from tensorflow import keras\n",
"from tensorflow.keras import layers\n",
"\n",
"inputs = keras.Input(shape=(sequence_length, raw_data.shape[-1]))\n",
"x = layers.Flatten()(inputs)\n",
"x = layers.Dense(16, activation=\"relu\")(x)\n",
"outputs = layers.Dense(1)(x)\n",
"model = keras.Model(inputs, outputs)\n",
"\n",
"callbacks = [\n",
" keras.callbacks.ModelCheckpoint(\"jena_dense.keras\",\n",
" save_best_only=True)\n",
"]\n",
"model.compile(optimizer=\"rmsprop\", loss=\"mse\", metrics=[\"mae\"])\n",
"history = model.fit(train_dataset,\n",
" epochs=10,\n",
" validation_data=val_dataset,\n",
" callbacks=callbacks)\n",
"\n",
"model = keras.models.load_model(\"jena_dense.keras\")\n",
"print(f\"Test MAE: {model.evaluate(test_dataset)[1]:.2f}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Plotting results**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"loss = history.history[\"mae\"]\n",
"val_loss = history.history[\"val_mae\"]\n",
"epochs = range(1, len(loss) + 1)\n",
"plt.figure()\n",
"plt.plot(epochs, loss, \"bo\", label=\"Training MAE\")\n",
"plt.plot(epochs, val_loss, \"b\", label=\"Validation MAE\")\n",
"plt.title(\"Training and validation MAE\")\n",
"plt.legend()\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Let's try a 1D convolutional model"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"inputs = keras.Input(shape=(sequence_length, raw_data.shape[-1]))\n",
"x = layers.Conv1D(8, 24, activation=\"relu\")(inputs)\n",
"x = layers.MaxPooling1D(2)(x)\n",
"x = layers.Conv1D(8, 12, activation=\"relu\")(x)\n",
"x = layers.MaxPooling1D(2)(x)\n",
"x = layers.Conv1D(8, 6, activation=\"relu\")(x)\n",
"x = layers.GlobalAveragePooling1D()(x)\n",
"outputs = layers.Dense(1)(x)\n",
"model = keras.Model(inputs, outputs)\n",
"\n",
"callbacks = [\n",
" keras.callbacks.ModelCheckpoint(\"jena_conv.keras\",\n",
" save_best_only=True)\n",
"]\n",
"model.compile(optimizer=\"rmsprop\", loss=\"mse\", metrics=[\"mae\"])\n",
"history = model.fit(train_dataset,\n",
" epochs=10,\n",
" validation_data=val_dataset,\n",
" callbacks=callbacks)\n",
"\n",
"model = keras.models.load_model(\"jena_conv.keras\")\n",
"print(f\"Test MAE: {model.evaluate(test_dataset)[1]:.2f}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### A first recurrent baseline"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**A simple LSTM-based model**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"inputs = keras.Input(shape=(sequence_length, raw_data.shape[-1]))\n",
"x = layers.LSTM(16)(inputs)\n",
"outputs = layers.Dense(1)(x)\n",
"model = keras.Model(inputs, outputs)\n",
"\n",
"callbacks = [\n",
" keras.callbacks.ModelCheckpoint(\"jena_lstm.keras\",\n",
" save_best_only=True)\n",
"]\n",
"model.compile(optimizer=\"rmsprop\", loss=\"mse\", metrics=[\"mae\"])\n",
"history = model.fit(train_dataset,\n",
" epochs=10,\n",
" validation_data=val_dataset,\n",
" callbacks=callbacks)\n",
"\n",
"model = keras.models.load_model(\"jena_lstm.keras\")\n",
"print(f\"Test MAE: {model.evaluate(test_dataset)[1]:.2f}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## Understanding recurrent neural networks"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**NumPy implementation of a simple RNN**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"import numpy as np\n",
"timesteps = 100\n",
"input_features = 32\n",
"output_features = 64\n",
"inputs = np.random.random((timesteps, input_features))\n",
"state_t = np.zeros((output_features,))\n",
"W = np.random.random((output_features, input_features))\n",
"U = np.random.random((output_features, output_features))\n",
"b = np.random.random((output_features,))\n",
"successive_outputs = []\n",
"for input_t in inputs:\n",
" output_t = np.tanh(np.dot(W, input_t) + np.dot(U, state_t) + b)\n",
" successive_outputs.append(output_t)\n",
" state_t = output_t\n",
"final_output_sequence = np.stack(successive_outputs, axis=0)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### A recurrent layer in Keras"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**An RNN layer that can process sequences of any length**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"num_features = 14\n",
"inputs = keras.Input(shape=(None, num_features))\n",
"outputs = layers.SimpleRNN(16)(inputs)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**An RNN layer that returns only its last output step**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"num_features = 14\n",
"steps = 120\n",
"inputs = keras.Input(shape=(steps, num_features))\n",
"outputs = layers.SimpleRNN(16, return_sequences=False)(inputs)\n",
"print(outputs.shape)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**An RNN layer that returns its full output sequence**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"num_features = 14\n",
"steps = 120\n",
"inputs = keras.Input(shape=(steps, num_features))\n",
"outputs = layers.SimpleRNN(16, return_sequences=True)(inputs)\n",
"print(outputs.shape)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Stacking RNN layers**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"inputs = keras.Input(shape=(steps, num_features))\n",
"x = layers.SimpleRNN(16, return_sequences=True)(inputs)\n",
"x = layers.SimpleRNN(16, return_sequences=True)(x)\n",
"outputs = layers.SimpleRNN(16)(x)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## Advanced use of recurrent neural networks"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Using recurrent dropout to fight overfitting"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Training and evaluating a dropout-regularized LSTM**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"inputs = keras.Input(shape=(sequence_length, raw_data.shape[-1]))\n",
"x = layers.LSTM(32, recurrent_dropout=0.25)(inputs)\n",
"x = layers.Dropout(0.5)(x)\n",
"outputs = layers.Dense(1)(x)\n",
"model = keras.Model(inputs, outputs)\n",
"\n",
"callbacks = [\n",
" keras.callbacks.ModelCheckpoint(\"jena_lstm_dropout.keras\",\n",
" save_best_only=True)\n",
"]\n",
"model.compile(optimizer=\"rmsprop\", loss=\"mse\", metrics=[\"mae\"])\n",
"history = model.fit(train_dataset,\n",
" epochs=50,\n",
" validation_data=val_dataset,\n",
" callbacks=callbacks)"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"inputs = keras.Input(shape=(sequence_length, num_features))\n",
"x = layers.LSTM(32, recurrent_dropout=0.2, unroll=True)(inputs)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Stacking recurrent layers"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Training and evaluating a dropout-regularized, stacked GRU model**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"inputs = keras.Input(shape=(sequence_length, raw_data.shape[-1]))\n",
"x = layers.GRU(32, recurrent_dropout=0.5, return_sequences=True)(inputs)\n",
"x = layers.GRU(32, recurrent_dropout=0.5)(x)\n",
"x = layers.Dropout(0.5)(x)\n",
"outputs = layers.Dense(1)(x)\n",
"model = keras.Model(inputs, outputs)\n",
"\n",
"callbacks = [\n",
" keras.callbacks.ModelCheckpoint(\"jena_stacked_gru_dropout.keras\",\n",
" save_best_only=True)\n",
"]\n",
"model.compile(optimizer=\"rmsprop\", loss=\"mse\", metrics=[\"mae\"])\n",
"history = model.fit(train_dataset,\n",
" epochs=50,\n",
" validation_data=val_dataset,\n",
" callbacks=callbacks)\n",
"model = keras.models.load_model(\"jena_stacked_gru_dropout.keras\")\n",
"print(f\"Test MAE: {model.evaluate(test_dataset)[1]:.2f}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Using bidirectional RNNs"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Training and evaluating a bidirectional LSTM**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"inputs = keras.Input(shape=(sequence_length, raw_data.shape[-1]))\n",
"x = layers.Bidirectional(layers.LSTM(16))(inputs)\n",
"outputs = layers.Dense(1)(x)\n",
"model = keras.Model(inputs, outputs)\n",
"\n",
"model.compile(optimizer=\"rmsprop\", loss=\"mse\", metrics=[\"mae\"])\n",
"history = model.fit(train_dataset,\n",
" epochs=10,\n",
" validation_data=val_dataset)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Going even further"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## Summary"
]
}
],
"metadata": {
"colab": {
"collapsed_sections": [],
"name": "chapter10_dl-for-timeseries.i",
"private_outputs": false,
"provenance": [],
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.0"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
second_edition/chapter11_part01_introduction.ipynb
+754
View File
@@ -0,0 +1,754 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"This is a companion notebook for the book [Deep Learning with Python, Second Edition](https://www.manning.com/books/deep-learning-with-python-second-edition?a_aid=keras&a_bid=76564dff). For readability, it only contains runnable code blocks and section titles, and omits everything else in the book: text paragraphs, figures, and pseudocode.\n\n**If you want to be able to follow what's going on, I recommend reading the notebook side by side with your copy of the book.**\n\nThis notebook was generated for TensorFlow 2.6."
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"# Deep learning for text"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## Natural-language processing: The bird's eye view"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## Preparing text data"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Text standardization"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Text splitting (tokenization)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Vocabulary indexing"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Using the TextVectorization layer"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"import string\n",
"\n",
"class Vectorizer:\n",
" def standardize(self, text):\n",
" text = text.lower()\n",
" return \"\".join(char for char in text if char not in string.punctuation)\n",
"\n",
" def tokenize(self, text):\n",
" text = self.standardize(text)\n",
" return text.split()\n",
"\n",
" def make_vocabulary(self, dataset):\n",
" self.vocabulary = {\"\": 0, \"[UNK]\": 1}\n",
" for text in dataset:\n",
" text = self.standardize(text)\n",
" tokens = self.tokenize(text)\n",
" for token in tokens:\n",
" if token not in self.vocabulary:\n",
" self.vocabulary[token] = len(self.vocabulary)\n",
" self.inverse_vocabulary = dict(\n",
" (v, k) for k, v in self.vocabulary.items())\n",
"\n",
" def encode(self, text):\n",
" text = self.standardize(text)\n",
" tokens = self.tokenize(text)\n",
" return [self.vocabulary.get(token, 1) for token in tokens]\n",
"\n",
" def decode(self, int_sequence):\n",
" return \" \".join(\n",
" self.inverse_vocabulary.get(i, \"[UNK]\") for i in int_sequence)\n",
"\n",
"vectorizer = Vectorizer()\n",
"dataset = [\n",
" \"I write, erase, rewrite\",\n",
" \"Erase again, and then\",\n",
" \"A poppy blooms.\",\n",
"]\n",
"vectorizer.make_vocabulary(dataset)"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"test_sentence = \"I write, rewrite, and still rewrite again\"\n",
"encoded_sentence = vectorizer.encode(test_sentence)\n",
"print(encoded_sentence)"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"decoded_sentence = vectorizer.decode(encoded_sentence)\n",
"print(decoded_sentence)"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"from tensorflow.keras.layers import TextVectorization\n",
"text_vectorization = TextVectorization(\n",
" output_mode=\"int\",\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"import re\n",
"import string\n",
"import tensorflow as tf\n",
"\n",
"def custom_standardization_fn(string_tensor):\n",
" lowercase_string = tf.strings.lower(string_tensor)\n",
" return tf.strings.regex_replace(\n",
" lowercase_string, f\"[{re.escape(string.punctuation)}]\", \"\")\n",
"\n",
"def custom_split_fn(string_tensor):\n",
" return tf.strings.split(string_tensor)\n",
"\n",
"text_vectorization = TextVectorization(\n",
" output_mode=\"int\",\n",
" standardize=custom_standardization_fn,\n",
" split=custom_split_fn,\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"dataset = [\n",
" \"I write, erase, rewrite\",\n",
" \"Erase again, and then\",\n",
" \"A poppy blooms.\",\n",
"]\n",
"text_vectorization.adapt(dataset)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Displaying the vocabulary**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"text_vectorization.get_vocabulary()"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"vocabulary = text_vectorization.get_vocabulary()\n",
"test_sentence = \"I write, rewrite, and still rewrite again\"\n",
"encoded_sentence = text_vectorization(test_sentence)\n",
"print(encoded_sentence)"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"inverse_vocab = dict(enumerate(vocabulary))\n",
"decoded_sentence = \" \".join(inverse_vocab[int(i)] for i in encoded_sentence)\n",
"print(decoded_sentence)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## Two approaches for representing groups of words: Sets and sequences"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Preparing the IMDB movie reviews data"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"!curl -O https://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz\n",
"!tar -xf aclImdb_v1.tar.gz"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"!rm -r aclImdb/train/unsup"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"!cat aclImdb/train/pos/4077_10.txt"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"import os, pathlib, shutil, random\n",
"\n",
"base_dir = pathlib.Path(\"aclImdb\")\n",
"val_dir = base_dir / \"val\"\n",
"train_dir = base_dir / \"train\"\n",
"for category in (\"neg\", \"pos\"):\n",
" os.makedirs(val_dir / category)\n",
" files = os.listdir(train_dir / category)\n",
" random.Random(1337).shuffle(files)\n",
" num_val_samples = int(0.2 * len(files))\n",
" val_files = files[-num_val_samples:]\n",
" for fname in val_files:\n",
" shutil.move(train_dir / category / fname,\n",
" val_dir / category / fname)"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"from tensorflow import keras\n",
"batch_size = 32\n",
"\n",
"train_ds = keras.utils.text_dataset_from_directory(\n",
" \"aclImdb/train\", batch_size=batch_size\n",
")\n",
"val_ds = keras.utils.text_dataset_from_directory(\n",
" \"aclImdb/val\", batch_size=batch_size\n",
")\n",
"test_ds = keras.utils.text_dataset_from_directory(\n",
" \"aclImdb/test\", batch_size=batch_size\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Displaying the shapes and dtypes of the first batch**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"for inputs, targets in train_ds:\n",
" print(\"inputs.shape:\", inputs.shape)\n",
" print(\"inputs.dtype:\", inputs.dtype)\n",
" print(\"targets.shape:\", targets.shape)\n",
" print(\"targets.dtype:\", targets.dtype)\n",
" print(\"inputs[0]:\", inputs[0])\n",
" print(\"targets[0]:\", targets[0])\n",
" break"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Processing words as a set: The bag-of-words approach"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### Single words (unigrams) with binary encoding"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Preprocessing our datasets with a `TextVectorization` layer**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"text_vectorization = TextVectorization(\n",
" max_tokens=20000,\n",
" output_mode=\"multi_hot\",\n",
")\n",
"text_only_train_ds = train_ds.map(lambda x, y: x)\n",
"text_vectorization.adapt(text_only_train_ds)\n",
"\n",
"binary_1gram_train_ds = train_ds.map(\n",
" lambda x, y: (text_vectorization(x), y),\n",
" num_parallel_calls=4)\n",
"binary_1gram_val_ds = val_ds.map(\n",
" lambda x, y: (text_vectorization(x), y),\n",
" num_parallel_calls=4)\n",
"binary_1gram_test_ds = test_ds.map(\n",
" lambda x, y: (text_vectorization(x), y),\n",
" num_parallel_calls=4)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Inspecting the output of our binary unigram dataset**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"for inputs, targets in binary_1gram_train_ds:\n",
" print(\"inputs.shape:\", inputs.shape)\n",
" print(\"inputs.dtype:\", inputs.dtype)\n",
" print(\"targets.shape:\", targets.shape)\n",
" print(\"targets.dtype:\", targets.dtype)\n",
" print(\"inputs[0]:\", inputs[0])\n",
" print(\"targets[0]:\", targets[0])\n",
" break"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Our model-building utility**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"from tensorflow import keras\n",
"from tensorflow.keras import layers\n",
"\n",
"def get_model(max_tokens=20000, hidden_dim=16):\n",
" inputs = keras.Input(shape=(max_tokens,))\n",
" x = layers.Dense(hidden_dim, activation=\"relu\")(inputs)\n",
" x = layers.Dropout(0.5)(x)\n",
" outputs = layers.Dense(1, activation=\"sigmoid\")(x)\n",
" model = keras.Model(inputs, outputs)\n",
" model.compile(optimizer=\"rmsprop\",\n",
" loss=\"binary_crossentropy\",\n",
" metrics=[\"accuracy\"])\n",
" return model"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Training and testing the binary unigram model**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"model = get_model()\n",
"model.summary()\n",
"callbacks = [\n",
" keras.callbacks.ModelCheckpoint(\"binary_1gram.keras\",\n",
" save_best_only=True)\n",
"]\n",
"model.fit(binary_1gram_train_ds.cache(),\n",
" validation_data=binary_1gram_val_ds.cache(),\n",
" epochs=10,\n",
" callbacks=callbacks)\n",
"model = keras.models.load_model(\"binary_1gram.keras\")\n",
"print(f\"Test acc: {model.evaluate(binary_1gram_test_ds)[1]:.3f}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### Bigrams with binary encoding"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Configuring the `TextVectorization` layer to return bigrams**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"text_vectorization = TextVectorization(\n",
" ngrams=2,\n",
" max_tokens=20000,\n",
" output_mode=\"multi_hot\",\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Training and testing the binary bigram model**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"text_vectorization.adapt(text_only_train_ds)\n",
"binary_2gram_train_ds = train_ds.map(\n",
" lambda x, y: (text_vectorization(x), y),\n",
" num_parallel_calls=4)\n",
"binary_2gram_val_ds = val_ds.map(\n",
" lambda x, y: (text_vectorization(x), y),\n",
" num_parallel_calls=4)\n",
"binary_2gram_test_ds = test_ds.map(\n",
" lambda x, y: (text_vectorization(x), y),\n",
" num_parallel_calls=4)\n",
"\n",
"model = get_model()\n",
"model.summary()\n",
"callbacks = [\n",
" keras.callbacks.ModelCheckpoint(\"binary_2gram.keras\",\n",
" save_best_only=True)\n",
"]\n",
"model.fit(binary_2gram_train_ds.cache(),\n",
" validation_data=binary_2gram_val_ds.cache(),\n",
" epochs=10,\n",
" callbacks=callbacks)\n",
"model = keras.models.load_model(\"binary_2gram.keras\")\n",
"print(f\"Test acc: {model.evaluate(binary_2gram_test_ds)[1]:.3f}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### Bigrams with TF-IDF encoding"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Configuring the `TextVectorization` layer to return token counts**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"text_vectorization = TextVectorization(\n",
" ngrams=2,\n",
" max_tokens=20000,\n",
" output_mode=\"count\"\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Configuring `TextVectorization` to return TF-IDF-weighted outputs**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"text_vectorization = TextVectorization(\n",
" ngrams=2,\n",
" max_tokens=20000,\n",
" output_mode=\"tf_idf\",\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Training and testing the TF-IDF bigram model**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"text_vectorization.adapt(text_only_train_ds)\n",
"\n",
"tfidf_2gram_train_ds = train_ds.map(\n",
" lambda x, y: (text_vectorization(x), y),\n",
" num_parallel_calls=4)\n",
"tfidf_2gram_val_ds = val_ds.map(\n",
" lambda x, y: (text_vectorization(x), y),\n",
" num_parallel_calls=4)\n",
"tfidf_2gram_test_ds = test_ds.map(\n",
" lambda x, y: (text_vectorization(x), y),\n",
" num_parallel_calls=4)\n",
"\n",
"model = get_model()\n",
"model.summary()\n",
"callbacks = [\n",
" keras.callbacks.ModelCheckpoint(\"tfidf_2gram.keras\",\n",
" save_best_only=True)\n",
"]\n",
"model.fit(tfidf_2gram_train_ds.cache(),\n",
" validation_data=tfidf_2gram_val_ds.cache(),\n",
" epochs=10,\n",
" callbacks=callbacks)\n",
"model = keras.models.load_model(\"tfidf_2gram.keras\")\n",
"print(f\"Test acc: {model.evaluate(tfidf_2gram_test_ds)[1]:.3f}\")"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"inputs = keras.Input(shape=(1,), dtype=\"string\")\n",
"processed_inputs = text_vectorization(inputs)\n",
"outputs = model(processed_inputs)\n",
"inference_model = keras.Model(inputs, outputs)"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"import tensorflow as tf\n",
"raw_text_data = tf.convert_to_tensor([\n",
" [\"That was an excellent movie, I loved it.\"],\n",
"])\n",
"predictions = inference_model(raw_text_data)\n",
"print(f\"{float(predictions[0] * 100):.2f} percent positive\")"
]
}
],
"metadata": {
"colab": {
"collapsed_sections": [],
"name": "chapter11_part01_introduction.i",
"private_outputs": false,
"provenance": [],
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.0"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
second_edition/chapter11_part02_sequence-models.ipynb
+478
View File
@@ -0,0 +1,478 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"This is a companion notebook for the book [Deep Learning with Python, Second Edition](https://www.manning.com/books/deep-learning-with-python-second-edition?a_aid=keras&a_bid=76564dff). For readability, it only contains runnable code blocks and section titles, and omits everything else in the book: text paragraphs, figures, and pseudocode.\n\n**If you want to be able to follow what's going on, I recommend reading the notebook side by side with your copy of the book.**\n\nThis notebook was generated for TensorFlow 2.6."
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Processing words as a sequence: The sequence model approach"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### A first practical example"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Downloading the data**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"!curl -O https://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz\n",
"!tar -xf aclImdb_v1.tar.gz\n",
"!rm -r aclImdb/train/unsup"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Preparing the data**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"import os, pathlib, shutil, random\n",
"from tensorflow import keras\n",
"batch_size = 32\n",
"base_dir = pathlib.Path(\"aclImdb\")\n",
"val_dir = base_dir / \"val\"\n",
"train_dir = base_dir / \"train\"\n",
"for category in (\"neg\", \"pos\"):\n",
" os.makedirs(val_dir / category)\n",
" files = os.listdir(train_dir / category)\n",
" random.Random(1337).shuffle(files)\n",
" num_val_samples = int(0.2 * len(files))\n",
" val_files = files[-num_val_samples:]\n",
" for fname in val_files:\n",
" shutil.move(train_dir / category / fname,\n",
" val_dir / category / fname)\n",
"\n",
"train_ds = keras.utils.text_dataset_from_directory(\n",
" \"aclImdb/train\", batch_size=batch_size\n",
")\n",
"val_ds = keras.utils.text_dataset_from_directory(\n",
" \"aclImdb/val\", batch_size=batch_size\n",
")\n",
"test_ds = keras.utils.text_dataset_from_directory(\n",
" \"aclImdb/test\", batch_size=batch_size\n",
")\n",
"text_only_train_ds = train_ds.map(lambda x, y: x)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Preparing integer sequence datasets**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"from tensorflow.keras import layers\n",
"\n",
"max_length = 600\n",
"max_tokens = 20000\n",
"text_vectorization = layers.TextVectorization(\n",
" max_tokens=max_tokens,\n",
" output_mode=\"int\",\n",
" output_sequence_length=max_length,\n",
")\n",
"text_vectorization.adapt(text_only_train_ds)\n",
"\n",
"int_train_ds = train_ds.map(\n",
" lambda x, y: (text_vectorization(x), y),\n",
" num_parallel_calls=4)\n",
"int_val_ds = val_ds.map(\n",
" lambda x, y: (text_vectorization(x), y),\n",
" num_parallel_calls=4)\n",
"int_test_ds = test_ds.map(\n",
" lambda x, y: (text_vectorization(x), y),\n",
" num_parallel_calls=4)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**A sequence model built on one-hot encoded vector sequences**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"import tensorflow as tf\n",
"inputs = keras.Input(shape=(None,), dtype=\"int64\")\n",
"embedded = tf.one_hot(inputs, depth=max_tokens)\n",
"x = layers.Bidirectional(layers.LSTM(32))(embedded)\n",
"x = layers.Dropout(0.5)(x)\n",
"outputs = layers.Dense(1, activation=\"sigmoid\")(x)\n",
"model = keras.Model(inputs, outputs)\n",
"model.compile(optimizer=\"rmsprop\",\n",
" loss=\"binary_crossentropy\",\n",
" metrics=[\"accuracy\"])\n",
"model.summary()"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Training a first basic sequence model**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"callbacks = [\n",
" keras.callbacks.ModelCheckpoint(\"one_hot_bidir_lstm.keras\",\n",
" save_best_only=True)\n",
"]\n",
"model.fit(int_train_ds, validation_data=int_val_ds, epochs=10, callbacks=callbacks)\n",
"model = keras.models.load_model(\"one_hot_bidir_lstm.keras\")\n",
"print(f\"Test acc: {model.evaluate(int_test_ds)[1]:.3f}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### Understanding word embeddings"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### Learning word embeddings with the Embedding layer"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Instantiating an `Embedding` layer**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"embedding_layer = layers.Embedding(input_dim=max_tokens, output_dim=256)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Model that uses an `Embedding` layer trained from scratch**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"inputs = keras.Input(shape=(None,), dtype=\"int64\")\n",
"embedded = layers.Embedding(input_dim=max_tokens, output_dim=256)(inputs)\n",
"x = layers.Bidirectional(layers.LSTM(32))(embedded)\n",
"x = layers.Dropout(0.5)(x)\n",
"outputs = layers.Dense(1, activation=\"sigmoid\")(x)\n",
"model = keras.Model(inputs, outputs)\n",
"model.compile(optimizer=\"rmsprop\",\n",
" loss=\"binary_crossentropy\",\n",
" metrics=[\"accuracy\"])\n",
"model.summary()\n",
"\n",
"callbacks = [\n",
" keras.callbacks.ModelCheckpoint(\"embeddings_bidir_gru.keras\",\n",
" save_best_only=True)\n",
"]\n",
"model.fit(int_train_ds, validation_data=int_val_ds, epochs=10, callbacks=callbacks)\n",
"model = keras.models.load_model(\"embeddings_bidir_gru.keras\")\n",
"print(f\"Test acc: {model.evaluate(int_test_ds)[1]:.3f}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### Understanding padding and masking"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Using an `Embedding` layer with masking enabled**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"inputs = keras.Input(shape=(None,), dtype=\"int64\")\n",
"embedded = layers.Embedding(\n",
" input_dim=max_tokens, output_dim=256, mask_zero=True)(inputs)\n",
"x = layers.Bidirectional(layers.LSTM(32))(embedded)\n",
"x = layers.Dropout(0.5)(x)\n",
"outputs = layers.Dense(1, activation=\"sigmoid\")(x)\n",
"model = keras.Model(inputs, outputs)\n",
"model.compile(optimizer=\"rmsprop\",\n",
" loss=\"binary_crossentropy\",\n",
" metrics=[\"accuracy\"])\n",
"model.summary()\n",
"\n",
"callbacks = [\n",
" keras.callbacks.ModelCheckpoint(\"embeddings_bidir_gru_with_masking.keras\",\n",
" save_best_only=True)\n",
"]\n",
"model.fit(int_train_ds, validation_data=int_val_ds, epochs=10, callbacks=callbacks)\n",
"model = keras.models.load_model(\"embeddings_bidir_gru_with_masking.keras\")\n",
"print(f\"Test acc: {model.evaluate(int_test_ds)[1]:.3f}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### Using pretrained word embeddings"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"!wget http://nlp.stanford.edu/data/glove.6B.zip\n",
"!unzip -q glove.6B.zip"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Parsing the GloVe word-embeddings file**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"import numpy as np\n",
"path_to_glove_file = \"glove.6B.100d.txt\"\n",
"\n",
"embeddings_index = {}\n",
"with open(path_to_glove_file) as f:\n",
" for line in f:\n",
" word, coefs = line.split(maxsplit=1)\n",
" coefs = np.fromstring(coefs, \"f\", sep=\" \")\n",
" embeddings_index[word] = coefs\n",
"\n",
"print(f\"Found {len(embeddings_index)} word vectors.\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Preparing the GloVe word-embeddings matrix**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"embedding_dim = 100\n",
"\n",
"vocabulary = text_vectorization.get_vocabulary()\n",
"word_index = dict(zip(vocabulary, range(len(vocabulary))))\n",
"\n",
"embedding_matrix = np.zeros((max_tokens, embedding_dim))\n",
"for word, i in word_index.items():\n",
" if i < max_tokens:\n",
" embedding_vector = embeddings_index.get(word)\n",
" if embedding_vector is not None:\n",
" embedding_matrix[i] = embedding_vector"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"embedding_layer = layers.Embedding(\n",
" max_tokens,\n",
" embedding_dim,\n",
" embeddings_initializer=keras.initializers.Constant(embedding_matrix),\n",
" trainable=False,\n",
" mask_zero=True,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Model that uses a pretrained Embedding layer**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"inputs = keras.Input(shape=(None,), dtype=\"int64\")\n",
"embedded = embedding_layer(inputs)\n",
"x = layers.Bidirectional(layers.LSTM(32))(embedded)\n",
"x = layers.Dropout(0.5)(x)\n",
"outputs = layers.Dense(1, activation=\"sigmoid\")(x)\n",
"model = keras.Model(inputs, outputs)\n",
"model.compile(optimizer=\"rmsprop\",\n",
" loss=\"binary_crossentropy\",\n",
" metrics=[\"accuracy\"])\n",
"model.summary()\n",
"\n",
"callbacks = [\n",
" keras.callbacks.ModelCheckpoint(\"glove_embeddings_sequence_model.keras\",\n",
" save_best_only=True)\n",
"]\n",
"model.fit(int_train_ds, validation_data=int_val_ds, epochs=10, callbacks=callbacks)\n",
"model = keras.models.load_model(\"glove_embeddings_sequence_model.keras\")\n",
"print(f\"Test acc: {model.evaluate(int_test_ds)[1]:.3f}\")"
]
}
],
"metadata": {
"colab": {
"collapsed_sections": [],
"name": "chapter11_part02_sequence-models.i",
"private_outputs": false,
"provenance": [],
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.0"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
second_edition/chapter11_part03_transformer.ipynb
+432
View File
@@ -0,0 +1,432 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"This is a companion notebook for the book [Deep Learning with Python, Second Edition](https://www.manning.com/books/deep-learning-with-python-second-edition?a_aid=keras&a_bid=76564dff). For readability, it only contains runnable code blocks and section titles, and omits everything else in the book: text paragraphs, figures, and pseudocode.\n\n**If you want to be able to follow what's going on, I recommend reading the notebook side by side with your copy of the book.**\n\nThis notebook was generated for TensorFlow 2.6."
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## The Transformer architecture"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Understanding self-attention"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### Generalized self-attention: the query-key-value model"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Multi-head attention"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### The Transformer encoder"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Getting the data**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"!curl -O https://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz\n",
"!tar -xf aclImdb_v1.tar.gz\n",
"!rm -r aclImdb/train/unsup"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Preparing the data**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"import os, pathlib, shutil, random\n",
"from tensorflow import keras\n",
"batch_size = 32\n",
"base_dir = pathlib.Path(\"aclImdb\")\n",
"val_dir = base_dir / \"val\"\n",
"train_dir = base_dir / \"train\"\n",
"for category in (\"neg\", \"pos\"):\n",
" os.makedirs(val_dir / category)\n",
" files = os.listdir(train_dir / category)\n",
" random.Random(1337).shuffle(files)\n",
" num_val_samples = int(0.2 * len(files))\n",
" val_files = files[-num_val_samples:]\n",
" for fname in val_files:\n",
" shutil.move(train_dir / category / fname,\n",
" val_dir / category / fname)\n",
"\n",
"train_ds = keras.utils.text_dataset_from_directory(\n",
" \"aclImdb/train\", batch_size=batch_size\n",
")\n",
"val_ds = keras.utils.text_dataset_from_directory(\n",
" \"aclImdb/val\", batch_size=batch_size\n",
")\n",
"test_ds = keras.utils.text_dataset_from_directory(\n",
" \"aclImdb/test\", batch_size=batch_size\n",
")\n",
"text_only_train_ds = train_ds.map(lambda x, y: x)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Vectorizing the data**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"from tensorflow.keras import layers\n",
"\n",
"max_length = 600\n",
"max_tokens = 20000\n",
"text_vectorization = layers.TextVectorization(\n",
" max_tokens=max_tokens,\n",
" output_mode=\"int\",\n",
" output_sequence_length=max_length,\n",
")\n",
"text_vectorization.adapt(text_only_train_ds)\n",
"\n",
"int_train_ds = train_ds.map(\n",
" lambda x, y: (text_vectorization(x), y),\n",
" num_parallel_calls=4)\n",
"int_val_ds = val_ds.map(\n",
" lambda x, y: (text_vectorization(x), y),\n",
" num_parallel_calls=4)\n",
"int_test_ds = test_ds.map(\n",
" lambda x, y: (text_vectorization(x), y),\n",
" num_parallel_calls=4)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Transformer encoder implemented as a subclassed `Layer`**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"import tensorflow as tf\n",
"from tensorflow import keras\n",
"from tensorflow.keras import layers\n",
"\n",
"class TransformerEncoder(layers.Layer):\n",
" def __init__(self, embed_dim, dense_dim, num_heads, **kwargs):\n",
" super().__init__(**kwargs)\n",
" self.embed_dim = embed_dim\n",
" self.dense_dim = dense_dim\n",
" self.num_heads = num_heads\n",
" self.attention = layers.MultiHeadAttention(\n",
" num_heads=num_heads, key_dim=embed_dim)\n",
" self.dense_proj = keras.Sequential(\n",
" [layers.Dense(dense_dim, activation=\"relu\"),\n",
" layers.Dense(embed_dim),]\n",
" )\n",
" self.layernorm_1 = layers.LayerNormalization()\n",
" self.layernorm_2 = layers.LayerNormalization()\n",
"\n",
" def call(self, inputs, mask=None):\n",
" if mask is not None:\n",
" mask = mask[:, tf.newaxis, :]\n",
" attention_output = self.attention(\n",
" inputs, inputs, attention_mask=mask)\n",
" proj_input = self.layernorm_1(inputs + attention_output)\n",
" proj_output = self.dense_proj(proj_input)\n",
" return self.layernorm_2(proj_input + proj_output)\n",
"\n",
" def get_config(self):\n",
" config = super().get_config()\n",
" config.update({\n",
" \"embed_dim\": self.embed_dim,\n",
" \"num_heads\": self.num_heads,\n",
" \"dense_dim\": self.dense_dim,\n",
" })\n",
" return config"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Using the Transformer encoder for text classification**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"vocab_size = 20000\n",
"embed_dim = 256\n",
"num_heads = 2\n",
"dense_dim = 32\n",
"\n",
"inputs = keras.Input(shape=(None,), dtype=\"int64\")\n",
"x = layers.Embedding(vocab_size, embed_dim)(inputs)\n",
"x = TransformerEncoder(embed_dim, dense_dim, num_heads)(x)\n",
"x = layers.GlobalMaxPooling1D()(x)\n",
"x = layers.Dropout(0.5)(x)\n",
"outputs = layers.Dense(1, activation=\"sigmoid\")(x)\n",
"model = keras.Model(inputs, outputs)\n",
"model.compile(optimizer=\"rmsprop\",\n",
" loss=\"binary_crossentropy\",\n",
" metrics=[\"accuracy\"])\n",
"model.summary()"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Training and evaluating the Transformer encoder based model**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"callbacks = [\n",
" keras.callbacks.ModelCheckpoint(\"transformer_encoder.keras\",\n",
" save_best_only=True)\n",
"]\n",
"model.fit(int_train_ds, validation_data=int_val_ds, epochs=20, callbacks=callbacks)\n",
"model = keras.models.load_model(\n",
" \"transformer_encoder.keras\",\n",
" custom_objects={\"TransformerEncoder\": TransformerEncoder})\n",
"print(f\"Test acc: {model.evaluate(int_test_ds)[1]:.3f}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### Using positional encoding to re-inject order information"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Implementing positional embedding as a subclassed layer**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"class PositionalEmbedding(layers.Layer):\n",
" def __init__(self, sequence_length, input_dim, output_dim, **kwargs):\n",
" super().__init__(**kwargs)\n",
" self.token_embeddings = layers.Embedding(\n",
" input_dim=input_dim, output_dim=output_dim)\n",
" self.position_embeddings = layers.Embedding(\n",
" input_dim=sequence_length, output_dim=output_dim)\n",
" self.sequence_length = sequence_length\n",
" self.input_dim = input_dim\n",
" self.output_dim = output_dim\n",
"\n",
" def call(self, inputs):\n",
" length = tf.shape(inputs)[-1]\n",
" positions = tf.range(start=0, limit=length, delta=1)\n",
" embedded_tokens = self.token_embeddings(inputs)\n",
" embedded_positions = self.position_embeddings(positions)\n",
" return embedded_tokens + embedded_positions\n",
"\n",
" def compute_mask(self, inputs, mask=None):\n",
" return tf.math.not_equal(inputs, 0)\n",
"\n",
" def get_config(self):\n",
" config = super().get_config()\n",
" config.update({\n",
" \"output_dim\": self.output_dim,\n",
" \"sequence_length\": self.sequence_length,\n",
" \"input_dim\": self.input_dim,\n",
" })\n",
" return config"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### Putting it all together: A text-classification Transformer"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Combining the Transformer encoder with positional embedding**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"vocab_size = 20000\n",
"sequence_length = 600\n",
"embed_dim = 256\n",
"num_heads = 2\n",
"dense_dim = 32\n",
"\n",
"inputs = keras.Input(shape=(None,), dtype=\"int64\")\n",
"x = PositionalEmbedding(sequence_length, vocab_size, embed_dim)(inputs)\n",
"x = TransformerEncoder(embed_dim, dense_dim, num_heads)(x)\n",
"x = layers.GlobalMaxPooling1D()(x)\n",
"x = layers.Dropout(0.5)(x)\n",
"outputs = layers.Dense(1, activation=\"sigmoid\")(x)\n",
"model = keras.Model(inputs, outputs)\n",
"model.compile(optimizer=\"rmsprop\",\n",
" loss=\"binary_crossentropy\",\n",
" metrics=[\"accuracy\"])\n",
"model.summary()\n",
"\n",
"callbacks = [\n",
" keras.callbacks.ModelCheckpoint(\"full_transformer_encoder.keras\",\n",
" save_best_only=True)\n",
"]\n",
"model.fit(int_train_ds, validation_data=int_val_ds, epochs=20, callbacks=callbacks)\n",
"model = keras.models.load_model(\n",
" \"full_transformer_encoder.keras\",\n",
" custom_objects={\"TransformerEncoder\": TransformerEncoder,\n",
" \"PositionalEmbedding\": PositionalEmbedding})\n",
"print(f\"Test acc: {model.evaluate(int_test_ds)[1]:.3f}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### When to use sequence models over bag-of-words models?"
]
}
],
"metadata": {
"colab": {
"collapsed_sections": [],
"name": "chapter11_part03_transformer.i",
"private_outputs": false,
"provenance": [],
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.0"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
second_edition/chapter11_part04_sequence-to-sequence-learning.ipynb
+625
View File
@@ -0,0 +1,625 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"This is a companion notebook for the book [Deep Learning with Python, Second Edition](https://www.manning.com/books/deep-learning-with-python-second-edition?a_aid=keras&a_bid=76564dff). For readability, it only contains runnable code blocks and section titles, and omits everything else in the book: text paragraphs, figures, and pseudocode.\n\n**If you want to be able to follow what's going on, I recommend reading the notebook side by side with your copy of the book.**\n\nThis notebook was generated for TensorFlow 2.6."
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## Beyond text classification: Sequence-to-sequence learning"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### A machine translation example"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"!wget http://storage.googleapis.com/download.tensorflow.org/data/spa-eng.zip\n",
"!unzip -q spa-eng.zip"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"text_file = \"spa-eng/spa.txt\"\n",
"with open(text_file) as f:\n",
" lines = f.read().split(\"\\n\")[:-1]\n",
"text_pairs = []\n",
"for line in lines:\n",
" english, spanish = line.split(\"\\t\")\n",
" spanish = \"[start] \" + spanish + \" [end]\"\n",
" text_pairs.append((english, spanish))"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"import random\n",
"print(random.choice(text_pairs))"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"import random\n",
"random.shuffle(text_pairs)\n",
"num_val_samples = int(0.15 * len(text_pairs))\n",
"num_train_samples = len(text_pairs) - 2 * num_val_samples\n",
"train_pairs = text_pairs[:num_train_samples]\n",
"val_pairs = text_pairs[num_train_samples:num_train_samples + num_val_samples]\n",
"test_pairs = text_pairs[num_train_samples + num_val_samples:]"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Vectorizing the English and Spanish text pairs**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"import tensorflow as tf\n",
"import string\n",
"import re\n",
"from tensorflow import keras\n",
"from tensorflow.keras import layers\n",
"\n",
"strip_chars = string.punctuation + \"\u00bf\"\n",
"strip_chars = strip_chars.replace(\"[\", \"\")\n",
"strip_chars = strip_chars.replace(\"]\", \"\")\n",
"\n",
"def custom_standardization(input_string):\n",
" lowercase = tf.strings.lower(input_string)\n",
" return tf.strings.regex_replace(\n",
" lowercase, f\"[{re.escape(strip_chars)}]\", \"\")\n",
"\n",
"vocab_size = 15000\n",
"sequence_length = 20\n",
"\n",
"source_vectorization = layers.TextVectorization(\n",
" max_tokens=vocab_size,\n",
" output_mode=\"int\",\n",
" output_sequence_length=sequence_length,\n",
")\n",
"target_vectorization = layers.TextVectorization(\n",
" max_tokens=vocab_size,\n",
" output_mode=\"int\",\n",
" output_sequence_length=sequence_length + 1,\n",
" standardize=custom_standardization,\n",
")\n",
"train_english_texts = [pair[0] for pair in train_pairs]\n",
"train_spanish_texts = [pair[1] for pair in train_pairs]\n",
"source_vectorization.adapt(train_english_texts)\n",
"target_vectorization.adapt(train_spanish_texts)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Preparing datasets for the translation task**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"batch_size = 64\n",
"\n",
"def format_dataset(eng, spa):\n",
" eng = source_vectorization(eng)\n",
" spa = target_vectorization(spa)\n",
" return ({\n",
" \"english\": eng,\n",
" \"spanish\": spa[:, :-1],\n",
" }, spa[:, 1:])\n",
"\n",
"def make_dataset(pairs):\n",
" eng_texts, spa_texts = zip(*pairs)\n",
" eng_texts = list(eng_texts)\n",
" spa_texts = list(spa_texts)\n",
" dataset = tf.data.Dataset.from_tensor_slices((eng_texts, spa_texts))\n",
" dataset = dataset.batch(batch_size)\n",
" dataset = dataset.map(format_dataset, num_parallel_calls=4)\n",
" return dataset.shuffle(2048).prefetch(16).cache()\n",
"\n",
"train_ds = make_dataset(train_pairs)\n",
"val_ds = make_dataset(val_pairs)"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"for inputs, targets in train_ds.take(1):\n",
" print(f\"inputs['english'].shape: {inputs['english'].shape}\")\n",
" print(f\"inputs['spanish'].shape: {inputs['spanish'].shape}\")\n",
" print(f\"targets.shape: {targets.shape}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Sequence-to-sequence learning with RNNs"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**GRU-based encoder**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"from tensorflow import keras\n",
"from tensorflow.keras import layers\n",
"\n",
"embed_dim = 256\n",
"latent_dim = 1024\n",
"\n",
"source = keras.Input(shape=(None,), dtype=\"int64\", name=\"english\")\n",
"x = layers.Embedding(vocab_size, embed_dim, mask_zero=True)(source)\n",
"encoded_source = layers.Bidirectional(\n",
" layers.GRU(latent_dim), merge_mode=\"sum\")(x)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**GRU-based decoder and the end-to-end model**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"past_target = keras.Input(shape=(None,), dtype=\"int64\", name=\"spanish\")\n",
"x = layers.Embedding(vocab_size, embed_dim, mask_zero=True)(past_target)\n",
"decoder_gru = layers.GRU(latent_dim, return_sequences=True)\n",
"x = decoder_gru(x, initial_state=encoded_source)\n",
"x = layers.Dropout(0.5)(x)\n",
"target_next_step = layers.Dense(vocab_size, activation=\"softmax\")(x)\n",
"seq2seq_rnn = keras.Model([source, past_target], target_next_step)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Training our recurrent sequence-to-sequence model**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"seq2seq_rnn.compile(\n",
" optimizer=\"rmsprop\",\n",
" loss=\"sparse_categorical_crossentropy\",\n",
" metrics=[\"accuracy\"])\n",
"seq2seq_rnn.fit(train_ds, epochs=15, validation_data=val_ds)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Translating new sentences with our RNN encoder and decoder**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"import numpy as np\n",
"spa_vocab = target_vectorization.get_vocabulary()\n",
"spa_index_lookup = dict(zip(range(len(spa_vocab)), spa_vocab))\n",
"max_decoded_sentence_length = 20\n",
"\n",
"def decode_sequence(input_sentence):\n",
" tokenized_input_sentence = source_vectorization([input_sentence])\n",
" decoded_sentence = \"[start]\"\n",
" for i in range(max_decoded_sentence_length):\n",
" tokenized_target_sentence = target_vectorization([decoded_sentence])\n",
" next_token_predictions = seq2seq_rnn.predict(\n",
" [tokenized_input_sentence, tokenized_target_sentence])\n",
" sampled_token_index = np.argmax(next_token_predictions[0, i, :])\n",
" sampled_token = spa_index_lookup[sampled_token_index]\n",
" decoded_sentence += \" \" + sampled_token\n",
" if sampled_token == \"[end]\":\n",
" break\n",
" return decoded_sentence\n",
"\n",
"test_eng_texts = [pair[0] for pair in test_pairs]\n",
"for _ in range(20):\n",
" input_sentence = random.choice(test_eng_texts)\n",
" print(\"-\")\n",
" print(input_sentence)\n",
" print(decode_sequence(input_sentence))"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Sequence-to-sequence learning with Transformer"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### The Transformer decoder"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**The `TransformerDecoder`**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"class TransformerDecoder(layers.Layer):\n",
" def __init__(self, embed_dim, dense_dim, num_heads, **kwargs):\n",
" super().__init__(**kwargs)\n",
" self.embed_dim = embed_dim\n",
" self.dense_dim = dense_dim\n",
" self.num_heads = num_heads\n",
" self.attention_1 = layers.MultiHeadAttention(\n",
" num_heads=num_heads, key_dim=embed_dim)\n",
" self.attention_2 = layers.MultiHeadAttention(\n",
" num_heads=num_heads, key_dim=embed_dim)\n",
" self.dense_proj = keras.Sequential(\n",
" [layers.Dense(dense_dim, activation=\"relu\"),\n",
" layers.Dense(embed_dim),]\n",
" )\n",
" self.layernorm_1 = layers.LayerNormalization()\n",
" self.layernorm_2 = layers.LayerNormalization()\n",
" self.layernorm_3 = layers.LayerNormalization()\n",
" self.supports_masking = True\n",
"\n",
" def get_config(self):\n",
" config = super().get_config()\n",
" config.update({\n",
" \"embed_dim\": self.embed_dim,\n",
" \"num_heads\": self.num_heads,\n",
" \"dense_dim\": self.dense_dim,\n",
" })\n",
" return config\n",
"\n",
" def get_causal_attention_mask(self, inputs):\n",
" input_shape = tf.shape(inputs)\n",
" batch_size, sequence_length = input_shape[0], input_shape[1]\n",
" i = tf.range(sequence_length)[:, tf.newaxis]\n",
" j = tf.range(sequence_length)\n",
" mask = tf.cast(i >= j, dtype=\"int32\")\n",
" mask = tf.reshape(mask, (1, input_shape[1], input_shape[1]))\n",
" mult = tf.concat(\n",
" [tf.expand_dims(batch_size, -1),\n",
" tf.constant([1, 1], dtype=tf.int32)], axis=0)\n",
" return tf.tile(mask, mult)\n",
"\n",
" def call(self, inputs, encoder_outputs, mask=None):\n",
" causal_mask = self.get_causal_attention_mask(inputs)\n",
" if mask is not None:\n",
" padding_mask = tf.cast(\n",
" mask[:, tf.newaxis, :], dtype=\"int32\")\n",
" padding_mask = tf.minimum(padding_mask, causal_mask)\n",
" else:\n",
" padding_mask = mask\n",
" attention_output_1 = self.attention_1(\n",
" query=inputs,\n",
" value=inputs,\n",
" key=inputs,\n",
" attention_mask=causal_mask)\n",
" attention_output_1 = self.layernorm_1(inputs + attention_output_1)\n",
" attention_output_2 = self.attention_2(\n",
" query=attention_output_1,\n",
" value=encoder_outputs,\n",
" key=encoder_outputs,\n",
" attention_mask=padding_mask,\n",
" )\n",
" attention_output_2 = self.layernorm_2(\n",
" attention_output_1 + attention_output_2)\n",
" proj_output = self.dense_proj(attention_output_2)\n",
" return self.layernorm_3(attention_output_2 + proj_output)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### Putting it all together: A Transformer for machine translation"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**PositionalEmbedding layer**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"class PositionalEmbedding(layers.Layer):\n",
" def __init__(self, sequence_length, input_dim, output_dim, **kwargs):\n",
" super().__init__(**kwargs)\n",
" self.token_embeddings = layers.Embedding(\n",
" input_dim=input_dim, output_dim=output_dim)\n",
" self.position_embeddings = layers.Embedding(\n",
" input_dim=sequence_length, output_dim=output_dim)\n",
" self.sequence_length = sequence_length\n",
" self.input_dim = input_dim\n",
" self.output_dim = output_dim\n",
"\n",
" def call(self, inputs):\n",
" length = tf.shape(inputs)[-1]\n",
" positions = tf.range(start=0, limit=length, delta=1)\n",
" embedded_tokens = self.token_embeddings(inputs)\n",
" embedded_positions = self.position_embeddings(positions)\n",
" return embedded_tokens + embedded_positions\n",
"\n",
" def compute_mask(self, inputs, mask=None):\n",
" return tf.math.not_equal(inputs, 0)\n",
"\n",
" def get_config(self):\n",
" config = super(PositionalEmbedding, self).get_config()\n",
" config.update({\n",
" \"output_dim\": self.output_dim,\n",
" \"sequence_length\": self.sequence_length,\n",
" \"input_dim\": self.input_dim,\n",
" })\n",
" return config"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**End-to-end Transformer**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"embed_dim = 256\n",
"dense_dim = 2048\n",
"num_heads = 8\n",
"\n",
"encoder_inputs = keras.Input(shape=(None,), dtype=\"int64\", name=\"english\")\n",
"x = PositionalEmbedding(sequence_length, vocab_size, embed_dim)(encoder_inputs)\n",
"encoder_outputs = TransformerEncoder(embed_dim, dense_dim, num_heads)(x)\n",
"\n",
"decoder_inputs = keras.Input(shape=(None,), dtype=\"int64\", name=\"spanish\")\n",
"x = PositionalEmbedding(sequence_length, vocab_size, embed_dim)(decoder_inputs)\n",
"x = TransformerDecoder(embed_dim, dense_dim, num_heads)(x, encoder_outputs)\n",
"x = layers.Dropout(0.5)(x)\n",
"decoder_outputs = layers.Dense(vocab_size, activation=\"softmax\")(x)\n",
"transformer = keras.Model([encoder_inputs, decoder_inputs], decoder_outputs)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Training the sequence-to-sequence Transformer**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"transformer.compile(\n",
" optimizer=\"rmsprop\",\n",
" loss=\"sparse_categorical_crossentropy\",\n",
" metrics=[\"accuracy\"])\n",
"transformer.fit(train_ds, epochs=30, validation_data=val_ds)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Translating new sentences with our Transformer model**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"import numpy as np\n",
"spa_vocab = target_vectorization.get_vocabulary()\n",
"spa_index_lookup = dict(zip(range(len(spa_vocab)), spa_vocab))\n",
"max_decoded_sentence_length = 20\n",
"\n",
"def decode_sequence(input_sentence):\n",
" tokenized_input_sentence = source_vectorization([input_sentence])\n",
" decoded_sentence = \"[start]\"\n",
" for i in range(max_decoded_sentence_length):\n",
" tokenized_target_sentence = target_vectorization(\n",
" [decoded_sentence])[:, :-1]\n",
" predictions = transformer(\n",
" [tokenized_input_sentence, tokenized_target_sentence])\n",
" sampled_token_index = np.argmax(predictions[0, i, :])\n",
" sampled_token = spa_index_lookup[sampled_token_index]\n",
" decoded_sentence += \" \" + sampled_token\n",
" if sampled_token == \"[end]\":\n",
" break\n",
" return decoded_sentence\n",
"\n",
"test_eng_texts = [pair[0] for pair in test_pairs]\n",
"for _ in range(20):\n",
" input_sentence = random.choice(test_eng_texts)\n",
" print(\"-\")\n",
" print(input_sentence)\n",
" print(decode_sequence(input_sentence))"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## Summary"
]
}
],
"metadata": {
"colab": {
"collapsed_sections": [],
"name": "chapter11_part04_sequence-to-sequence-learning.i",
"private_outputs": false,
"provenance": [],
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.0"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
second_edition/chapter12_part01_text-generation.ipynb
+481
View File
@@ -0,0 +1,481 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"This is a companion notebook for the book [Deep Learning with Python, Second Edition](https://www.manning.com/books/deep-learning-with-python-second-edition?a_aid=keras&a_bid=76564dff). For readability, it only contains runnable code blocks and section titles, and omits everything else in the book: text paragraphs, figures, and pseudocode.\n\n**If you want to be able to follow what's going on, I recommend reading the notebook side by side with your copy of the book.**\n\nThis notebook was generated for TensorFlow 2.6."
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"# Generative deep learning"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## Text generation"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### A brief history of generative deep learning for sequence generation"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### How do you generate sequence data?"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### The importance of the sampling strategy"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Reweighting a probability distribution to a different temperature**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"import numpy as np\n",
"def reweight_distribution(original_distribution, temperature=0.5):\n",
" distribution = np.log(original_distribution) / temperature\n",
" distribution = np.exp(distribution)\n",
" return distribution / np.sum(distribution)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Implementing text generation with Keras"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### Preparing the data"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Downloading and uncompressing the IMDB movie reviews dataset**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"!wget https://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz\n",
"!tar -xf aclImdb_v1.tar.gz"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Creating a dataset from text files (one file = one sample)**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"import tensorflow as tf\n",
"from tensorflow import keras\n",
"dataset = keras.utils.text_dataset_from_directory(\n",
" directory=\"aclImdb\", label_mode=None, batch_size=256)\n",
"dataset = dataset.map(lambda x: tf.strings.regex_replace(x, \"<br />\", \" \"))"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Preparing a `TextVectorization` layer**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"from tensorflow.keras.layers import TextVectorization\n",
"\n",
"sequence_length = 100\n",
"vocab_size = 15000\n",
"text_vectorization = TextVectorization(\n",
" max_tokens=vocab_size,\n",
" output_mode=\"int\",\n",
" output_sequence_length=sequence_length,\n",
")\n",
"text_vectorization.adapt(dataset)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Setting up a language modeling dataset**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"def prepare_lm_dataset(text_batch):\n",
" vectorized_sequences = text_vectorization(text_batch)\n",
" x = vectorized_sequences[:, :-1]\n",
" y = vectorized_sequences[:, 1:]\n",
" return x, y\n",
"\n",
"lm_dataset = dataset.map(prepare_lm_dataset, num_parallel_calls=4)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### A Transformer-based sequence-to-sequence model"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"import tensorflow as tf\n",
"from tensorflow.keras import layers\n",
"\n",
"class PositionalEmbedding(layers.Layer):\n",
" def __init__(self, sequence_length, input_dim, output_dim, **kwargs):\n",
" super().__init__(**kwargs)\n",
" self.token_embeddings = layers.Embedding(\n",
" input_dim=input_dim, output_dim=output_dim)\n",
" self.position_embeddings = layers.Embedding(\n",
" input_dim=sequence_length, output_dim=output_dim)\n",
" self.sequence_length = sequence_length\n",
" self.input_dim = input_dim\n",
" self.output_dim = output_dim\n",
"\n",
" def call(self, inputs):\n",
" length = tf.shape(inputs)[-1]\n",
" positions = tf.range(start=0, limit=length, delta=1)\n",
" embedded_tokens = self.token_embeddings(inputs)\n",
" embedded_positions = self.position_embeddings(positions)\n",
" return embedded_tokens + embedded_positions\n",
"\n",
" def compute_mask(self, inputs, mask=None):\n",
" return tf.math.not_equal(inputs, 0)\n",
"\n",
" def get_config(self):\n",
" config = super(PositionalEmbedding, self).get_config()\n",
" config.update({\n",
" \"output_dim\": self.output_dim,\n",
" \"sequence_length\": self.sequence_length,\n",
" \"input_dim\": self.input_dim,\n",
" })\n",
" return config\n",
"\n",
"\n",
"class TransformerDecoder(layers.Layer):\n",
" def __init__(self, embed_dim, dense_dim, num_heads, **kwargs):\n",
" super().__init__(**kwargs)\n",
" self.embed_dim = embed_dim\n",
" self.dense_dim = dense_dim\n",
" self.num_heads = num_heads\n",
" self.attention_1 = layers.MultiHeadAttention(\n",
" num_heads=num_heads, key_dim=embed_dim)\n",
" self.attention_2 = layers.MultiHeadAttention(\n",
" num_heads=num_heads, key_dim=embed_dim)\n",
" self.dense_proj = keras.Sequential(\n",
" [layers.Dense(dense_dim, activation=\"relu\"),\n",
" layers.Dense(embed_dim),]\n",
" )\n",
" self.layernorm_1 = layers.LayerNormalization()\n",
" self.layernorm_2 = layers.LayerNormalization()\n",
" self.layernorm_3 = layers.LayerNormalization()\n",
" self.supports_masking = True\n",
"\n",
" def get_config(self):\n",
" config = super(TransformerDecoder, self).get_config()\n",
" config.update({\n",
" \"embed_dim\": self.embed_dim,\n",
" \"num_heads\": self.num_heads,\n",
" \"dense_dim\": self.dense_dim,\n",
" })\n",
" return config\n",
"\n",
" def get_causal_attention_mask(self, inputs):\n",
" input_shape = tf.shape(inputs)\n",
" batch_size, sequence_length = input_shape[0], input_shape[1]\n",
" i = tf.range(sequence_length)[:, tf.newaxis]\n",
" j = tf.range(sequence_length)\n",
" mask = tf.cast(i >= j, dtype=\"int32\")\n",
" mask = tf.reshape(mask, (1, input_shape[1], input_shape[1]))\n",
" mult = tf.concat(\n",
" [tf.expand_dims(batch_size, -1),\n",
" tf.constant([1, 1], dtype=tf.int32)], axis=0)\n",
" return tf.tile(mask, mult)\n",
"\n",
" def call(self, inputs, encoder_outputs, mask=None):\n",
" causal_mask = self.get_causal_attention_mask(inputs)\n",
" if mask is not None:\n",
" padding_mask = tf.cast(\n",
" mask[:, tf.newaxis, :], dtype=\"int32\")\n",
" padding_mask = tf.minimum(padding_mask, causal_mask)\n",
" else:\n",
" padding_mask = mask\n",
" attention_output_1 = self.attention_1(\n",
" query=inputs,\n",
" value=inputs,\n",
" key=inputs,\n",
" attention_mask=causal_mask)\n",
" attention_output_1 = self.layernorm_1(inputs + attention_output_1)\n",
" attention_output_2 = self.attention_2(\n",
" query=attention_output_1,\n",
" value=encoder_outputs,\n",
" key=encoder_outputs,\n",
" attention_mask=padding_mask,\n",
" )\n",
" attention_output_2 = self.layernorm_2(\n",
" attention_output_1 + attention_output_2)\n",
" proj_output = self.dense_proj(attention_output_2)\n",
" return self.layernorm_3(attention_output_2 + proj_output)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**A simple Transformer-based language model**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"from tensorflow.keras import layers\n",
"embed_dim = 256\n",
"latent_dim = 2048\n",
"num_heads = 2\n",
"\n",
"inputs = keras.Input(shape=(None,), dtype=\"int64\")\n",
"x = PositionalEmbedding(sequence_length, vocab_size, embed_dim)(inputs)\n",
"x = TransformerDecoder(embed_dim, latent_dim, num_heads)(x, x)\n",
"outputs = layers.Dense(vocab_size, activation=\"softmax\")(x)\n",
"model = keras.Model(inputs, outputs)\n",
"model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"rmsprop\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### A text-generation callback with variable-temperature sampling"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**The text-generation callback**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"import numpy as np\n",
"\n",
"tokens_index = dict(enumerate(text_vectorization.get_vocabulary()))\n",
"\n",
"def sample_next(predictions, temperature=1.0):\n",
" predictions = np.asarray(predictions).astype(\"float64\")\n",
" predictions = np.log(predictions) / temperature\n",
" exp_preds = np.exp(predictions)\n",
" predictions = exp_preds / np.sum(exp_preds)\n",
" probas = np.random.multinomial(1, predictions, 1)\n",
" return np.argmax(probas)\n",
"\n",
"class TextGenerator(keras.callbacks.Callback):\n",
" def __init__(self,\n",
" prompt,\n",
" generate_length,\n",
" model_input_length,\n",
" temperatures=(1.,),\n",
" print_freq=1):\n",
" self.prompt = prompt\n",
" self.generate_length = generate_length\n",
" self.model_input_length = model_input_length\n",
" self.temperatures = temperatures\n",
" self.print_freq = print_freq\n",
" vectorized_prompt = text_vectorization([prompt])[0].numpy()\n",
" self.prompt_length = np.nonzero(vectorized_prompt == 0)[0][0]\n",
"\n",
" def on_epoch_end(self, epoch, logs=None):\n",
" if (epoch + 1) % self.print_freq != 0:\n",
" return\n",
" for temperature in self.temperatures:\n",
" print(\"== Generating with temperature\", temperature)\n",
" sentence = self.prompt\n",
" for i in range(self.generate_length):\n",
" tokenized_sentence = text_vectorization([sentence])\n",
" predictions = self.model(tokenized_sentence)\n",
" next_token = sample_next(\n",
" predictions[0, self.prompt_length - 1 + i, :]\n",
" )\n",
" sampled_token = tokens_index[next_token]\n",
" sentence += \" \" + sampled_token\n",
" print(sentence)\n",
"\n",
"prompt = \"This movie\"\n",
"text_gen_callback = TextGenerator(\n",
" prompt,\n",
" generate_length=50,\n",
" model_input_length=sequence_length,\n",
" temperatures=(0.2, 0.5, 0.7, 1., 1.5))"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Fitting the language model**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"model.fit(lm_dataset, epochs=200, callbacks=[text_gen_callback])"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Wrapping up"
]
}
],
"metadata": {
"colab": {
"collapsed_sections": [],
"name": "chapter12_part01_text-generation.i",
"private_outputs": false,
"provenance": [],
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.0"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
second_edition/chapter12_part02_deep-dream.ipynb
+308
View File
@@ -0,0 +1,308 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"This is a companion notebook for the book [Deep Learning with Python, Second Edition](https://www.manning.com/books/deep-learning-with-python-second-edition?a_aid=keras&a_bid=76564dff). For readability, it only contains runnable code blocks and section titles, and omits everything else in the book: text paragraphs, figures, and pseudocode.\n\n**If you want to be able to follow what's going on, I recommend reading the notebook side by side with your copy of the book.**\n\nThis notebook was generated for TensorFlow 2.6."
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## DeepDream"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Implementing DeepDream in Keras"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Fetching the test image**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"from tensorflow import keras\n",
"import matplotlib.pyplot as plt\n",
"\n",
"base_image_path = keras.utils.get_file(\n",
" \"coast.jpg\", origin=\"https://img-datasets.s3.amazonaws.com/coast.jpg\")\n",
"\n",
"plt.axis(\"off\")\n",
"plt.imshow(keras.utils.load_img(base_image_path))"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Instantiating a pretrained `InceptionV3` model**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"from tensorflow.keras.applications import inception_v3\n",
"model = inception_v3.InceptionV3(weights=\"imagenet\", include_top=False)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Configuring the contribution of each layer to the DeepDream loss**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"layer_settings = {\n",
" \"mixed4\": 1.0,\n",
" \"mixed5\": 1.5,\n",
" \"mixed6\": 2.0,\n",
" \"mixed7\": 2.5,\n",
"}\n",
"outputs_dict = dict(\n",
" [\n",
" (layer.name, layer.output)\n",
" for layer in [model.get_layer(name) for name in layer_settings.keys()]\n",
" ]\n",
")\n",
"feature_extractor = keras.Model(inputs=model.inputs, outputs=outputs_dict)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**The DeepDream loss**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"def compute_loss(input_image):\n",
" features = feature_extractor(input_image)\n",
" loss = tf.zeros(shape=())\n",
" for name in features.keys():\n",
" coeff = layer_settings[name]\n",
" activation = features[name]\n",
" loss += coeff * tf.reduce_mean(tf.square(activation[:, 2:-2, 2:-2, :]))\n",
" return loss"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**The DeepDream gradient ascent process**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"import tensorflow as tf\n",
"\n",
"@tf.function\n",
"def gradient_ascent_step(image, learning_rate):\n",
" with tf.GradientTape() as tape:\n",
" tape.watch(image)\n",
" loss = compute_loss(image)\n",
" grads = tape.gradient(loss, image)\n",
" grads = tf.math.l2_normalize(grads)\n",
" image += learning_rate * grads\n",
" return loss, image\n",
"\n",
"\n",
"def gradient_ascent_loop(image, iterations, learning_rate, max_loss=None):\n",
" for i in range(iterations):\n",
" loss, image = gradient_ascent_step(image, learning_rate)\n",
" if max_loss is not None and loss > max_loss:\n",
" break\n",
" print(f\"... Loss value at step {i}: {loss:.2f}\")\n",
" return image"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"step = 20.\n",
"num_octave = 3\n",
"octave_scale = 1.4\n",
"iterations = 30\n",
"max_loss = 15."
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Image processing utilities**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"import numpy as np\n",
"\n",
"def preprocess_image(image_path):\n",
" img = keras.utils.load_img(image_path)\n",
" img = keras.utils.img_to_array(img)\n",
" img = np.expand_dims(img, axis=0)\n",
" img = keras.applications.inception_v3.preprocess_input(img)\n",
" return img\n",
"\n",
"def deprocess_image(img):\n",
" img = img.reshape((img.shape[1], img.shape[2], 3))\n",
" img /= 2.0\n",
" img += 0.5\n",
" img *= 255.\n",
" img = np.clip(img, 0, 255).astype(\"uint8\")\n",
" return img"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Running gradient ascent over multiple successive \"octaves\"**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"original_img = preprocess_image(base_image_path)\n",
"original_shape = original_img.shape[1:3]\n",
"\n",
"successive_shapes = [original_shape]\n",
"for i in range(1, num_octave):\n",
" shape = tuple([int(dim / (octave_scale ** i)) for dim in original_shape])\n",
" successive_shapes.append(shape)\n",
"successive_shapes = successive_shapes[::-1]\n",
"\n",
"shrunk_original_img = tf.image.resize(original_img, successive_shapes[0])\n",
"\n",
"img = tf.identity(original_img)\n",
"for i, shape in enumerate(successive_shapes):\n",
" print(f\"Processing octave {i} with shape {shape}\")\n",
" img = tf.image.resize(img, shape)\n",
" img = gradient_ascent_loop(\n",
" img, iterations=iterations, learning_rate=step, max_loss=max_loss\n",
" )\n",
" upscaled_shrunk_original_img = tf.image.resize(shrunk_original_img, shape)\n",
" same_size_original = tf.image.resize(original_img, shape)\n",
" lost_detail = same_size_original - upscaled_shrunk_original_img\n",
" img += lost_detail\n",
" shrunk_original_img = tf.image.resize(original_img, shape)\n",
"\n",
"keras.utils.save_img(\"dream.png\", deprocess_image(img.numpy()))"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Wrapping up"
]
}
],
"metadata": {
"colab": {
"collapsed_sections": [],
"name": "chapter12_part02_deep-dream.i",
"private_outputs": false,
"provenance": [],
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.0"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
second_edition/chapter12_part03_neural-style-transfer.ipynb
+356
View File
@@ -0,0 +1,356 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"This is a companion notebook for the book [Deep Learning with Python, Second Edition](https://www.manning.com/books/deep-learning-with-python-second-edition?a_aid=keras&a_bid=76564dff). For readability, it only contains runnable code blocks and section titles, and omits everything else in the book: text paragraphs, figures, and pseudocode.\n\n**If you want to be able to follow what's going on, I recommend reading the notebook side by side with your copy of the book.**\n\nThis notebook was generated for TensorFlow 2.6."
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## Neural style transfer"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### The content loss"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### The style loss"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Neural style transfer in Keras"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Getting the style and content images**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"from tensorflow import keras\n",
"\n",
"base_image_path = keras.utils.get_file(\n",
" \"sf.jpg\", origin=\"https://img-datasets.s3.amazonaws.com/sf.jpg\")\n",
"style_reference_image_path = keras.utils.get_file(\n",
" \"starry_night.jpg\", origin=\"https://img-datasets.s3.amazonaws.com/starry_night.jpg\")\n",
"\n",
"original_width, original_height = keras.utils.load_img(base_image_path).size\n",
"img_height = 400\n",
"img_width = round(original_width * img_height / original_height)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Auxiliary functions**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"import numpy as np\n",
"\n",
"def preprocess_image(image_path):\n",
" img = keras.utils.load_img(\n",
" image_path, target_size=(img_height, img_width))\n",
" img = keras.utils.img_to_array(img)\n",
" img = np.expand_dims(img, axis=0)\n",
" img = keras.applications.vgg19.preprocess_input(img)\n",
" return img\n",
"\n",
"def deprocess_image(img):\n",
" img = img.reshape((img_height, img_width, 3))\n",
" img[:, :, 0] += 103.939\n",
" img[:, :, 1] += 116.779\n",
" img[:, :, 2] += 123.68\n",
" img = img[:, :, ::-1]\n",
" img = np.clip(img, 0, 255).astype(\"uint8\")\n",
" return img"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Using a pretrained VGG19 model to create a feature extractor**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"model = keras.applications.vgg19.VGG19(weights=\"imagenet\", include_top=False)\n",
"\n",
"outputs_dict = dict([(layer.name, layer.output) for layer in model.layers])\n",
"feature_extractor = keras.Model(inputs=model.inputs, outputs=outputs_dict)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Content loss**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"def content_loss(base_img, combination_img):\n",
" return tf.reduce_sum(tf.square(combination_img - base_img))"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Style loss**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"def gram_matrix(x):\n",
" x = tf.transpose(x, (2, 0, 1))\n",
" features = tf.reshape(x, (tf.shape(x)[0], -1))\n",
" gram = tf.matmul(features, tf.transpose(features))\n",
" return gram\n",
"\n",
"def style_loss(style_img, combination_img):\n",
" S = gram_matrix(style_img)\n",
" C = gram_matrix(combination_img)\n",
" channels = 3\n",
" size = img_height * img_width\n",
" return tf.reduce_sum(tf.square(S - C)) / (4.0 * (channels ** 2) * (size ** 2))"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Total variation loss**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"def total_variation_loss(x):\n",
" a = tf.square(\n",
" x[:, : img_height - 1, : img_width - 1, :] - x[:, 1:, : img_width - 1, :]\n",
" )\n",
" b = tf.square(\n",
" x[:, : img_height - 1, : img_width - 1, :] - x[:, : img_height - 1, 1:, :]\n",
" )\n",
" return tf.reduce_sum(tf.pow(a + b, 1.25))"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Defining the final loss that you'll minimize**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"style_layer_names = [\n",
" \"block1_conv1\",\n",
" \"block2_conv1\",\n",
" \"block3_conv1\",\n",
" \"block4_conv1\",\n",
" \"block5_conv1\",\n",
"]\n",
"content_layer_name = \"block5_conv2\"\n",
"total_variation_weight = 1e-6\n",
"style_weight = 1e-6\n",
"content_weight = 2.5e-8\n",
"\n",
"def compute_loss(combination_image, base_image, style_reference_image):\n",
" input_tensor = tf.concat(\n",
" [base_image, style_reference_image, combination_image], axis=0\n",
" )\n",
" features = feature_extractor(input_tensor)\n",
" loss = tf.zeros(shape=())\n",
" layer_features = features[content_layer_name]\n",
" base_image_features = layer_features[0, :, :, :]\n",
" combination_features = layer_features[2, :, :, :]\n",
" loss = loss + content_weight * content_loss(\n",
" base_image_features, combination_features\n",
" )\n",
" for layer_name in style_layer_names:\n",
" layer_features = features[layer_name]\n",
" style_reference_features = layer_features[1, :, :, :]\n",
" combination_features = layer_features[2, :, :, :]\n",
" style_loss_value = style_loss(\n",
" style_reference_features, combination_features)\n",
" loss += (style_weight / len(style_layer_names)) * style_loss_value\n",
"\n",
" loss += total_variation_weight * total_variation_loss(combination_image)\n",
" return loss"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Setting up the gradient-descent process**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"import tensorflow as tf\n",
"\n",
"@tf.function\n",
"def compute_loss_and_grads(combination_image, base_image, style_reference_image):\n",
" with tf.GradientTape() as tape:\n",
" loss = compute_loss(combination_image, base_image, style_reference_image)\n",
" grads = tape.gradient(loss, combination_image)\n",
" return loss, grads\n",
"\n",
"optimizer = keras.optimizers.SGD(\n",
" keras.optimizers.schedules.ExponentialDecay(\n",
" initial_learning_rate=100.0, decay_steps=100, decay_rate=0.96\n",
" )\n",
")\n",
"\n",
"base_image = preprocess_image(base_image_path)\n",
"style_reference_image = preprocess_image(style_reference_image_path)\n",
"combination_image = tf.Variable(preprocess_image(base_image_path))\n",
"\n",
"iterations = 4000\n",
"for i in range(1, iterations + 1):\n",
" loss, grads = compute_loss_and_grads(\n",
" combination_image, base_image, style_reference_image\n",
" )\n",
" optimizer.apply_gradients([(grads, combination_image)])\n",
" if i % 100 == 0:\n",
" print(f\"Iteration {i}: loss={loss:.2f}\")\n",
" img = deprocess_image(combination_image.numpy())\n",
" fname = f\"combination_image_at_iteration_{i}.png\"\n",
" keras.utils.save_img(fname, img)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Wrapping up"
]
}
],
"metadata": {
"colab": {
"collapsed_sections": [],
"name": "chapter12_part03_neural-style-transfer.i",
"private_outputs": false,
"provenance": [],
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.0"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
second_edition/chapter12_part04_variational-autoencoders.ipynb
+339
View File
@@ -0,0 +1,339 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"This is a companion notebook for the book [Deep Learning with Python, Second Edition](https://www.manning.com/books/deep-learning-with-python-second-edition?a_aid=keras&a_bid=76564dff). For readability, it only contains runnable code blocks and section titles, and omits everything else in the book: text paragraphs, figures, and pseudocode.\n\n**If you want to be able to follow what's going on, I recommend reading the notebook side by side with your copy of the book.**\n\nThis notebook was generated for TensorFlow 2.6."
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## Generating images with variational autoencoders"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Sampling from latent spaces of images"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Concept vectors for image editing"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Variational autoencoders"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Implementing a VAE with Keras"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**VAE encoder network**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"from tensorflow import keras\n",
"from tensorflow.keras import layers\n",
"\n",
"latent_dim = 2\n",
"\n",
"encoder_inputs = keras.Input(shape=(28, 28, 1))\n",
"x = layers.Conv2D(32, 3, activation=\"relu\", strides=2, padding=\"same\")(encoder_inputs)\n",
"x = layers.Conv2D(64, 3, activation=\"relu\", strides=2, padding=\"same\")(x)\n",
"x = layers.Flatten()(x)\n",
"x = layers.Dense(16, activation=\"relu\")(x)\n",
"z_mean = layers.Dense(latent_dim, name=\"z_mean\")(x)\n",
"z_log_var = layers.Dense(latent_dim, name=\"z_log_var\")(x)\n",
"encoder = keras.Model(encoder_inputs, [z_mean, z_log_var], name=\"encoder\")"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"encoder.summary()"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Latent-space-sampling layer**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"import tensorflow as tf\n",
"\n",
"class Sampler(layers.Layer):\n",
" def call(self, z_mean, z_log_var):\n",
" batch_size = tf.shape(z_mean)[0]\n",
" z_size = tf.shape(z_mean)[1]\n",
" epsilon = tf.random.normal(shape=(batch_size, z_size))\n",
" return z_mean + tf.exp(0.5 * z_log_var) * epsilon"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**VAE decoder network, mapping latent space points to images**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"latent_inputs = keras.Input(shape=(latent_dim,))\n",
"x = layers.Dense(7 * 7 * 64, activation=\"relu\")(latent_inputs)\n",
"x = layers.Reshape((7, 7, 64))(x)\n",
"x = layers.Conv2DTranspose(64, 3, activation=\"relu\", strides=2, padding=\"same\")(x)\n",
"x = layers.Conv2DTranspose(32, 3, activation=\"relu\", strides=2, padding=\"same\")(x)\n",
"decoder_outputs = layers.Conv2D(1, 3, activation=\"sigmoid\", padding=\"same\")(x)\n",
"decoder = keras.Model(latent_inputs, decoder_outputs, name=\"decoder\")"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"decoder.summary()"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**VAE model with custom `train_step()`**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"class VAE(keras.Model):\n",
" def __init__(self, encoder, decoder, **kwargs):\n",
" super().__init__(**kwargs)\n",
" self.encoder = encoder\n",
" self.decoder = decoder\n",
" self.sampler = Sampler()\n",
" self.total_loss_tracker = keras.metrics.Mean(name=\"total_loss\")\n",
" self.reconstruction_loss_tracker = keras.metrics.Mean(\n",
" name=\"reconstruction_loss\")\n",
" self.kl_loss_tracker = keras.metrics.Mean(name=\"kl_loss\")\n",
"\n",
" @property\n",
" def metrics(self):\n",
" return [self.total_loss_tracker,\n",
" self.reconstruction_loss_tracker,\n",
" self.kl_loss_tracker]\n",
"\n",
" def train_step(self, data):\n",
" with tf.GradientTape() as tape:\n",
" z_mean, z_log_var = self.encoder(data)\n",
" z = self.sampler(z_mean, z_log_var)\n",
" reconstruction = decoder(z)\n",
" reconstruction_loss = tf.reduce_mean(\n",
" tf.reduce_sum(\n",
" keras.losses.binary_crossentropy(data, reconstruction),\n",
" axis=(1, 2)\n",
" )\n",
" )\n",
" kl_loss = -0.5 * (1 + z_log_var - tf.square(z_mean) - tf.exp(z_log_var))\n",
" total_loss = reconstruction_loss + tf.reduce_mean(kl_loss)\n",
" grads = tape.gradient(total_loss, self.trainable_weights)\n",
" self.optimizer.apply_gradients(zip(grads, self.trainable_weights))\n",
" self.total_loss_tracker.update_state(total_loss)\n",
" self.reconstruction_loss_tracker.update_state(reconstruction_loss)\n",
" self.kl_loss_tracker.update_state(kl_loss)\n",
" return {\n",
" \"total_loss\": self.total_loss_tracker.result(),\n",
" \"reconstruction_loss\": self.reconstruction_loss_tracker.result(),\n",
" \"kl_loss\": self.kl_loss_tracker.result(),\n",
" }"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Training the VAE**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"import numpy as np\n",
"\n",
"(x_train, _), (x_test, _) = keras.datasets.mnist.load_data()\n",
"mnist_digits = np.concatenate([x_train, x_test], axis=0)\n",
"mnist_digits = np.expand_dims(mnist_digits, -1).astype(\"float32\") / 255\n",
"\n",
"vae = VAE(encoder, decoder)\n",
"vae.compile(optimizer=keras.optimizers.Adam(), run_eagerly=True)\n",
"vae.fit(mnist_digits, epochs=30, batch_size=128)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Sampling a grid of images from the 2D latent space**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"\n",
"n = 30\n",
"digit_size = 28\n",
"figure = np.zeros((digit_size * n, digit_size * n))\n",
"\n",
"grid_x = np.linspace(-1, 1, n)\n",
"grid_y = np.linspace(-1, 1, n)[::-1]\n",
"\n",
"for i, yi in enumerate(grid_y):\n",
" for j, xi in enumerate(grid_x):\n",
" z_sample = np.array([[xi, yi]])\n",
" x_decoded = vae.decoder.predict(z_sample)\n",
" digit = x_decoded[0].reshape(digit_size, digit_size)\n",
" figure[\n",
" i * digit_size : (i + 1) * digit_size,\n",
" j * digit_size : (j + 1) * digit_size,\n",
" ] = digit\n",
"\n",
"plt.figure(figsize=(15, 15))\n",
"start_range = digit_size // 2\n",
"end_range = n * digit_size + start_range\n",
"pixel_range = np.arange(start_range, end_range, digit_size)\n",
"sample_range_x = np.round(grid_x, 1)\n",
"sample_range_y = np.round(grid_y, 1)\n",
"plt.xticks(pixel_range, sample_range_x)\n",
"plt.yticks(pixel_range, sample_range_y)\n",
"plt.xlabel(\"z[0]\")\n",
"plt.ylabel(\"z[1]\")\n",
"plt.axis(\"off\")\n",
"plt.imshow(figure, cmap=\"Greys_r\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Wrapping up"
]
}
],
"metadata": {
"colab": {
"collapsed_sections": [],
"name": "chapter12_part04_variational-autoencoders.i",
"private_outputs": false,
"provenance": [],
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.0"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
second_edition/chapter12_part05_gans.ipynb
+447
View File
@@ -0,0 +1,447 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"This is a companion notebook for the book [Deep Learning with Python, Second Edition](https://www.manning.com/books/deep-learning-with-python-second-edition?a_aid=keras&a_bid=76564dff). For readability, it only contains runnable code blocks and section titles, and omits everything else in the book: text paragraphs, figures, and pseudocode.\n\n**If you want to be able to follow what's going on, I recommend reading the notebook side by side with your copy of the book.**\n\nThis notebook was generated for TensorFlow 2.6."
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## Introduction to generative adversarial networks"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### A schematic GAN implementation"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### A bag of tricks"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Getting our hands on the CelebA dataset"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Getting the CelebA data**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"!mkdir celeba_gan\n",
"!gdown --id 1O7m1010EJjLE5QxLZiM9Fpjs7Oj6e684 -O celeba_gan/data.zip\n",
"!unzip -qq celeba_gan/data.zip -d celeba_gan"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Creating a dataset from a directory of images**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"from tensorflow import keras\n",
"dataset = keras.utils.image_dataset_from_directory(\n",
" \"celeba_gan\",\n",
" label_mode=None,\n",
" image_size=(64, 64),\n",
" batch_size=32,\n",
" smart_resize=True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Rescaling the images**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"dataset = dataset.map(lambda x: x / 255.)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Displaying the first image**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"for x in dataset:\n",
" plt.axis(\"off\")\n",
" plt.imshow((x.numpy() * 255).astype(\"int32\")[0])\n",
" break"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### The discriminator"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**The GAN discriminator network**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"from tensorflow.keras import layers\n",
"\n",
"discriminator = keras.Sequential(\n",
" [\n",
" keras.Input(shape=(64, 64, 3)),\n",
" layers.Conv2D(64, kernel_size=4, strides=2, padding=\"same\"),\n",
" layers.LeakyReLU(alpha=0.2),\n",
" layers.Conv2D(128, kernel_size=4, strides=2, padding=\"same\"),\n",
" layers.LeakyReLU(alpha=0.2),\n",
" layers.Conv2D(128, kernel_size=4, strides=2, padding=\"same\"),\n",
" layers.LeakyReLU(alpha=0.2),\n",
" layers.Flatten(),\n",
" layers.Dropout(0.2),\n",
" layers.Dense(1, activation=\"sigmoid\"),\n",
" ],\n",
" name=\"discriminator\",\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"discriminator.summary()"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### The generator"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**GAN generator network**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"latent_dim = 128\n",
"\n",
"generator = keras.Sequential(\n",
" [\n",
" keras.Input(shape=(latent_dim,)),\n",
" layers.Dense(8 * 8 * 128),\n",
" layers.Reshape((8, 8, 128)),\n",
" layers.Conv2DTranspose(128, kernel_size=4, strides=2, padding=\"same\"),\n",
" layers.LeakyReLU(alpha=0.2),\n",
" layers.Conv2DTranspose(256, kernel_size=4, strides=2, padding=\"same\"),\n",
" layers.LeakyReLU(alpha=0.2),\n",
" layers.Conv2DTranspose(512, kernel_size=4, strides=2, padding=\"same\"),\n",
" layers.LeakyReLU(alpha=0.2),\n",
" layers.Conv2D(3, kernel_size=5, padding=\"same\", activation=\"sigmoid\"),\n",
" ],\n",
" name=\"generator\",\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"generator.summary()"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### The adversarial network"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**The GAN `Model`**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"import tensorflow as tf\n",
"class GAN(keras.Model):\n",
" def __init__(self, discriminator, generator, latent_dim):\n",
" super().__init__()\n",
" self.discriminator = discriminator\n",
" self.generator = generator\n",
" self.latent_dim = latent_dim\n",
" self.d_loss_metric = keras.metrics.Mean(name=\"d_loss\")\n",
" self.g_loss_metric = keras.metrics.Mean(name=\"g_loss\")\n",
"\n",
" def compile(self, d_optimizer, g_optimizer, loss_fn):\n",
" super(GAN, self).compile()\n",
" self.d_optimizer = d_optimizer\n",
" self.g_optimizer = g_optimizer\n",
" self.loss_fn = loss_fn\n",
"\n",
" @property\n",
" def metrics(self):\n",
" return [self.d_loss_metric, self.g_loss_metric]\n",
"\n",
" def train_step(self, real_images):\n",
" batch_size = tf.shape(real_images)[0]\n",
" random_latent_vectors = tf.random.normal(\n",
" shape=(batch_size, self.latent_dim))\n",
" generated_images = self.generator(random_latent_vectors)\n",
" combined_images = tf.concat([generated_images, real_images], axis=0)\n",
" labels = tf.concat(\n",
" [tf.ones((batch_size, 1)), tf.zeros((batch_size, 1))],\n",
" axis=0\n",
" )\n",
" labels += 0.05 * tf.random.uniform(tf.shape(labels))\n",
"\n",
" with tf.GradientTape() as tape:\n",
" predictions = self.discriminator(combined_images)\n",
" d_loss = self.loss_fn(labels, predictions)\n",
" grads = tape.gradient(d_loss, self.discriminator.trainable_weights)\n",
" self.d_optimizer.apply_gradients(\n",
" zip(grads, self.discriminator.trainable_weights)\n",
" )\n",
"\n",
" random_latent_vectors = tf.random.normal(\n",
" shape=(batch_size, self.latent_dim))\n",
"\n",
" misleading_labels = tf.zeros((batch_size, 1))\n",
"\n",
" with tf.GradientTape() as tape:\n",
" predictions = self.discriminator(\n",
" self.generator(random_latent_vectors))\n",
" g_loss = self.loss_fn(misleading_labels, predictions)\n",
" grads = tape.gradient(g_loss, self.generator.trainable_weights)\n",
" self.g_optimizer.apply_gradients(\n",
" zip(grads, self.generator.trainable_weights))\n",
"\n",
" self.d_loss_metric.update_state(d_loss)\n",
" self.g_loss_metric.update_state(g_loss)\n",
" return {\"d_loss\": self.d_loss_metric.result(),\n",
" \"g_loss\": self.g_loss_metric.result()}"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**A callback that samples generated images during training**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"class GANMonitor(keras.callbacks.Callback):\n",
" def __init__(self, num_img=3, latent_dim=128):\n",
" self.num_img = num_img\n",
" self.latent_dim = latent_dim\n",
"\n",
" def on_epoch_end(self, epoch, logs=None):\n",
" random_latent_vectors = tf.random.normal(shape=(self.num_img, self.latent_dim))\n",
" generated_images = self.model.generator(random_latent_vectors)\n",
" generated_images *= 255\n",
" generated_images.numpy()\n",
" for i in range(self.num_img):\n",
" img = keras.utils.array_to_img(generated_images[i])\n",
" img.save(f\"generated_img_{epoch:03d}_{i}.png\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Compiling and training the GAN**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"epochs = 100\n",
"\n",
"gan = GAN(discriminator=discriminator, generator=generator, latent_dim=latent_dim)\n",
"gan.compile(\n",
" d_optimizer=keras.optimizers.Adam(learning_rate=0.0001),\n",
" g_optimizer=keras.optimizers.Adam(learning_rate=0.0001),\n",
" loss_fn=keras.losses.BinaryCrossentropy(),\n",
")\n",
"\n",
"gan.fit(\n",
" dataset, epochs=epochs, callbacks=[GANMonitor(num_img=10, latent_dim=latent_dim)]\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Wrapping up"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## Summary"
]
}
],
"metadata": {
"colab": {
"collapsed_sections": [],
"name": "chapter12_part05_gans.i",
"private_outputs": false,
"provenance": [],
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.0"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
second_edition/chapter13_best-practices-for-the-real-world.ipynb
+466
View File
@@ -0,0 +1,466 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"This is a companion notebook for the book [Deep Learning with Python, Second Edition](https://www.manning.com/books/deep-learning-with-python-second-edition?a_aid=keras&a_bid=76564dff). For readability, it only contains runnable code blocks and section titles, and omits everything else in the book: text paragraphs, figures, and pseudocode.\n\n**If you want to be able to follow what's going on, I recommend reading the notebook side by side with your copy of the book.**\n\nThis notebook was generated for TensorFlow 2.6."
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"# Best practices for the real world"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## Getting the most out of your models"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Hyperparameter optimization"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### Using KerasTuner"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"!pip install keras-tuner -q"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**A KerasTuner model-building function**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"from tensorflow import keras\n",
"from tensorflow.keras import layers\n",
"\n",
"def build_model(hp):\n",
" units = hp.Int(name=\"units\", min_value=16, max_value=64, step=16)\n",
" model = keras.Sequential([\n",
" layers.Dense(units, activation=\"relu\"),\n",
" layers.Dense(10, activation=\"softmax\")\n",
" ])\n",
" optimizer = hp.Choice(name=\"optimizer\", values=[\"rmsprop\", \"adam\"])\n",
" model.compile(\n",
" optimizer=optimizer,\n",
" loss=\"sparse_categorical_crossentropy\",\n",
" metrics=[\"accuracy\"])\n",
" return model"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**A KerasTuner `HyperModel`**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"import kerastuner as kt\n",
"\n",
"class SimpleMLP(kt.HyperModel):\n",
" def __init__(self, num_classes):\n",
" self.num_classes = num_classes\n",
"\n",
" def build(self, hp):\n",
" units = hp.Int(name=\"units\", min_value=16, max_value=64, step=16)\n",
" model = keras.Sequential([\n",
" layers.Dense(units, activation=\"relu\"),\n",
" layers.Dense(self.num_classes, activation=\"softmax\")\n",
" ])\n",
" optimizer = hp.Choice(name=\"optimizer\", values=[\"rmsprop\", \"adam\"])\n",
" model.compile(\n",
" optimizer=optimizer,\n",
" loss=\"sparse_categorical_crossentropy\",\n",
" metrics=[\"accuracy\"])\n",
" return model\n",
"\n",
"hypermodel = SimpleMLP(num_classes=10)"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"tuner = kt.BayesianOptimization(\n",
" build_model,\n",
" objective=\"val_accuracy\",\n",
" max_trials=100,\n",
" executions_per_trial=2,\n",
" directory=\"mnist_kt_test\",\n",
" overwrite=True,\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"tuner.search_space_summary()"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()\n",
"x_train = x_train.reshape((-1, 28 * 28)).astype(\"float32\") / 255\n",
"x_test = x_test.reshape((-1, 28 * 28)).astype(\"float32\") / 255\n",
"x_train_full = x_train[:]\n",
"y_train_full = y_train[:]\n",
"num_val_samples = 10000\n",
"x_train, x_val = x_train[:-num_val_samples], x_train[-num_val_samples:]\n",
"y_train, y_val = y_train[:-num_val_samples], y_train[-num_val_samples:]\n",
"callbacks = [\n",
" keras.callbacks.EarlyStopping(monitor=\"val_loss\", patience=5),\n",
"]\n",
"tuner.search(\n",
" x_train, y_train,\n",
" batch_size=128,\n",
" epochs=100,\n",
" validation_data=(x_val, y_val),\n",
" callbacks=callbacks,\n",
" verbose=2,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"**Querying the best hyperparameter configurations**"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"top_n = 4\n",
"best_hps = tuner.get_best_hyperparameters(top_n)"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"def get_best_epoch(hp):\n",
" model = build_model(hp)\n",
" callbacks=[\n",
" keras.callbacks.EarlyStopping(\n",
" monitor=\"val_loss\", mode=\"min\", patience=10)\n",
" ]\n",
" history = model.fit(\n",
" x_train, y_train,\n",
" validation_data=(x_val, y_val),\n",
" epochs=100,\n",
" batch_size=128,\n",
" callbacks=callbacks)\n",
" val_loss_per_epoch = history.history[\"val_loss\"]\n",
" best_epoch = val_loss_per_epoch.index(min(val_loss_per_epoch)) + 1\n",
" print(f\"Best epoch: {best_epoch}\")\n",
" return best_epoch"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"def get_best_trained_model(hp):\n",
" best_epoch = get_best_epoch(hp)\n",
" model = build_model(hp)\n",
" model.fit(\n",
" x_train_full, y_train_full,\n",
" batch_size=128, epochs=int(best_epoch * 1.2))\n",
" return model\n",
"\n",
"best_models = []\n",
"for hp in best_hps:\n",
" model = get_best_trained_model(hp)\n",
" model.evaluate(x_test, y_test)\n",
" best_models.append(model)"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"best_models = tuner.get_best_models(top_n)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### The art of crafting the right search space"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### The future of hyperparameter tuning: automated machine learning"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Model ensembling"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## Scaling-up model training"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Speeding up training on GPU with mixed precision"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### Understanding floating-point precision"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"import tensorflow as tf\n",
"import numpy as np\n",
"np_array = np.zeros((2, 2))\n",
"tf_tensor = tf.convert_to_tensor(np_array)\n",
"tf_tensor.dtype"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"np_array = np.zeros((2, 2))\n",
"tf_tensor = tf.convert_to_tensor(np_array, dtype=\"float32\")\n",
"tf_tensor.dtype"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### Mixed-precision training in practice"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"from tensorflow import keras\n",
"keras.mixed_precision.set_global_policy(\"mixed_float16\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Multi-GPU training"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### Getting your hands on two or more GPUs"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### Single-host, multi-device synchronous training"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### TPU training"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### Using a TPU via Google Colab"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### Leveraging step fusing to improve TPU utilization"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## Summary"
]
}
],
"metadata": {
"colab": {
"collapsed_sections": [],
"name": "chapter13_best-practices-for-the-real-world.i",
"private_outputs": false,
"provenance": [],
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.0"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
second_edition/chapter14_conclusions.ipynb
+568
View File
@@ -0,0 +1,568 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"This is a companion notebook for the book [Deep Learning with Python, Second Edition](https://www.manning.com/books/deep-learning-with-python-second-edition?a_aid=keras&a_bid=76564dff). For readability, it only contains runnable code blocks and section titles, and omits everything else in the book: text paragraphs, figures, and pseudocode.\n\n**If you want to be able to follow what's going on, I recommend reading the notebook side by side with your copy of the book.**\n\nThis notebook was generated for TensorFlow 2.6."
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"# Conclusions"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## Key concepts in review"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Various approaches to AI"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### What makes deep learning special within the field of machine learning"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### How to think about deep learning"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Key enabling technologies"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### The universal machine-learning workflow"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Key network architectures"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### Densely connected networks"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"from tensorflow import keras\n",
"from tensorflow.keras\u00a0import\u00a0layers\n",
"inputs = keras.Input(shape=(num_input_features,))\n",
"x = layers.Dense(32,\u00a0activation=\"relu\")(inputs)\n",
"x = layers.Dense(32,\u00a0activation=\"relu\")(x)\n",
"outputs = layers.Dense(1,\u00a0activation=\"sigmoid\")(x)\n",
"model = keras.Model(inputs, outputs)\n",
"model.compile(optimizer=\"rmsprop\",\u00a0loss=\"binary_crossentropy\")"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"inputs = keras.Input(shape=(num_input_features,))\n",
"x = layers.Dense(32,\u00a0activation=\"relu\")(inputs)\n",
"x = layers.Dense(32,\u00a0activation=\"relu\")(x)\n",
"outputs = layers.Dense(num_classes,\u00a0activation=\"softmax\")(x)\n",
"model = keras.Model(inputs, outputs)\n",
"model.compile(optimizer=\"rmsprop\",\u00a0loss=\"categorical_crossentropy\")"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"inputs = keras.Input(shape=(num_input_features,))\n",
"x = layers.Dense(32,\u00a0activation=\"relu\")(inputs)\n",
"x = layers.Dense(32,\u00a0activation=\"relu\")(x)\n",
"outputs = layers.Dense(num_classes,\u00a0activation=\"sigmoid\")(x)\n",
"model = keras.Model(inputs, outputs)\n",
"model.compile(optimizer=\"rmsprop\",\u00a0loss=\"binary_crossentropy\")"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"inputs = keras.Input(shape=(num_input_features,))\n",
"x = layers.Dense(32,\u00a0activation=\"relu\")(inputs)\n",
"x = layers.Dense(32,\u00a0activation=\"relu\")(x)\n",
"outputs layers.Dense(num_values)(x)\n",
"model = keras.Model(inputs, outputs)\n",
"model.compile(optimizer=\"rmsprop\",\u00a0loss=\"mse\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### Convnets"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"inputs = keras.Input(shape=(height,\u00a0width,\u00a0channels))\n",
"x = layers.SeparableConv2D(32,\u00a03,\u00a0activation=\"relu\")(inputs)\n",
"x = layers.SeparableConv2D(64,\u00a03,\u00a0activation=\"relu\")(x)\n",
"x = layers.MaxPooling2D(2)(x)\n",
"x = layers.SeparableConv2D(64,\u00a03,\u00a0activation=\"relu\")(x)\n",
"x = layers.SeparableConv2D(128,\u00a03,\u00a0activation=\"relu\")(x)\n",
"x = layers.MaxPooling2D(2)(x)\n",
"x = layers.SeparableConv2D(64,\u00a03,\u00a0activation=\"relu\")(x)\n",
"x = layers.SeparableConv2D(128,\u00a03,\u00a0activation=\"relu\")(x)\n",
"x = layers.GlobalAveragePooling2D()(x)\n",
"x = layers.Dense(32,\u00a0activation=\"relu\")(x)\n",
"outputs = layers.Dense(num_classes,\u00a0activation=\"softmax\")(x)\n",
"model = keras.Model(inputs, outputs)\n",
"model.compile(optimizer=\"rmsprop\",\u00a0loss=\"categorical_crossentropy\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### RNNs"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"inputs = keras.Input(shape=(num_timesteps,\u00a0num_features))\n",
"x = layers.LSTM(32)(inputs)\n",
"outputs = layers.Dense(num_classes,\u00a0activation=\"sigmoid\")(x)\n",
"model = keras.Model(inputs, outputs)\n",
"model.compile(optimizer=\"rmsprop\",\u00a0loss=\"binary_crossentropy\")"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"inputs = keras.Input(shape=(num_timesteps,\u00a0num_features))\n",
"x = layers.LSTM(32,\u00a0return_sequences=True)(inputs)\n",
"x = layers.LSTM(32,\u00a0return_sequences=True)(x)\n",
"x = layers.LSTM(32)(x)\n",
"outputs = layers.Dense(num_classes,\u00a0activation=\"sigmoid\")(x)\n",
"model = keras.Model(inputs, outputs)\n",
"model.compile(optimizer=\"rmsprop\",\u00a0loss=\"binary_crossentropy\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### Transformers"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"encoder_inputs = keras.Input(shape=(sequence_length,), dtype=\"int64\")\n",
"x = PositionalEmbedding(sequence_length, vocab_size, embed_dim)(encoder_inputs)\n",
"encoder_outputs = TransformerEncoder(embed_dim, dense_dim, num_heads)(x)\n",
"decoder_inputs = keras.Input(shape=(None,), dtype=\"int64\")\n",
"x = PositionalEmbedding(sequence_length, vocab_size, embed_dim)(decoder_inputs)\n",
"x = TransformerDecoder(embed_dim, dense_dim, num_heads)(x, encoder_outputs)\n",
"decoder_outputs = layers.Dense(vocab_size, activation=\"softmax\")(x)\n",
"transformer = keras.Model([encoder_inputs, decoder_inputs], decoder_outputs)\n",
"transformer.compile(optimizer=\"rmsprop\", loss=\"categorical_crossentropy\")"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab_type": "code"
},
"outputs": [],
"source": [
"inputs = keras.Input(shape=(sequence_length,), dtype=\"int64\")\n",
"x = PositionalEmbedding(sequence_length, vocab_size, embed_dim)(inputs)\n",
"x = TransformerEncoder(embed_dim, dense_dim, num_heads)(x)\n",
"x = layers.GlobalMaxPooling1D()(x)\n",
"outputs = layers.Dense(1, activation=\"sigmoid\")(x)\n",
"model = keras.Model(inputs, outputs)\n",
"model.compile(optimizer=\"rmsprop\", loss=\"binary_crossentropy\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### The space of possibilities"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## The limitations of deep learning"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### The risk of anthropomorphizing machine-learning models"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Automatons vs. intelligent agents"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Local generalization vs. extreme generalization"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### The purpose of intelligence"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Climbing the spectrum of generalization"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## Setting the course toward greater generality in AI"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### On the importance of setting the right objective: The shortcut rule"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### A new target"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## Implementing intelligence: The missing ingredients"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Intelligence as sensitivity to abstract analogies"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### The two poles of abstraction"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### Value-centric analogy"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### Program-centric analogy"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### Cognition as a combination of both kinds of abstraction"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### The missing half of the picture"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## The future of deep learning"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Models as programs"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Blending together deep learning and program synthesis"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### Integrating deep-learning modules and algorithmic modules into hybrid systems"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"#### Using deep learning to guide program search"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Lifelong learning and modular subroutine reuse"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### The long-term vision"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## Staying up to date in a fast-moving field"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Practice on real-world problems using Kaggle"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Read about the latest developments on arXiv"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"### Explore the Keras ecosystem"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text"
},
"source": [
"## Final words"
]
}
],
"metadata": {
"colab": {
"collapsed_sections": [],
"name": "chapter14_conclusions.i",
"private_outputs": false,
"provenance": [],
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.0"
}
},
"nbformat": 4,
"nbformat_minor": 0
}