wehub/fchollet--deep-learning-with-python-notebooks
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 13:25:23 +08:00
commit 612b28061b
60 changed files with 49225 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
first_edition/2.1-a-first-look-at-a-neural-network.ipynb
+451
View File
@@ -0,0 +1,451 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Using TensorFlow backend.\n"
]
},
{
"data": {
"text/plain": [
"'2.0.8'"
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import keras\n",
"keras.__version__"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# A first look at a neural network\n",
"\n",
"This notebook contains the code samples found in Chapter 2, Section 1 of [Deep Learning with Python](https://www.manning.com/books/deep-learning-with-python?a_aid=keras&a_bid=76564dff). Note that the original text features far more content, in particular further explanations and figures: in this notebook, you will only find source code and related comments.\n",
"\n",
"----\n",
"\n",
"We will now take a look at a first concrete example of a neural network, which makes use of the Python library Keras to learn to classify \n",
"hand-written digits. Unless you already have experience with Keras or similar libraries, you will not understand everything about this \n",
"first example right away. You probably haven't even installed Keras yet. Don't worry, that is perfectly fine. In the next chapter, we will \n",
"review each element in our example and explain them in detail. So don't worry if some steps seem arbitrary or look like magic to you! \n",
"We've got to start somewhere.\n",
"\n",
"The problem we are trying to solve here is to classify grayscale images of handwritten digits (28 pixels by 28 pixels), into their 10 \n",
"categories (0 to 9). The dataset we will use is the MNIST dataset, a classic dataset in the machine learning community, which has been \n",
"around for almost as long as the field itself and has been very intensively studied. It's a set of 60,000 training images, plus 10,000 test \n",
"images, assembled by the National Institute of Standards and Technology (the NIST in MNIST) in the 1980s. You can think of \"solving\" MNIST \n",
"as the \"Hello World\" of deep learning -- it's what you do to verify that your algorithms are working as expected. As you become a machine \n",
"learning practitioner, you will see MNIST come up over and over again, in scientific papers, blog posts, and so on."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The MNIST dataset comes pre-loaded in Keras, in the form of a set of four Numpy arrays:"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"from keras.datasets import mnist\n",
"\n",
"(train_images, train_labels), (test_images, test_labels) = mnist.load_data()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"`train_images` and `train_labels` form the \"training set\", the data that the model will learn from. The model will then be tested on the \n",
"\"test set\", `test_images` and `test_labels`. Our images are encoded as Numpy arrays, and the labels are simply an array of digits, ranging \n",
"from 0 to 9. There is a one-to-one correspondence between the images and the labels.\n",
"\n",
"Let's have a look at the training data:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(60000, 28, 28)"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"train_images.shape"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"60000"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"len(train_labels)"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([5, 0, 4, ..., 5, 6, 8], dtype=uint8)"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"train_labels"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's have a look at the test data:"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(10000, 28, 28)"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"test_images.shape"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"10000"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"len(test_labels)"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([7, 2, 1, ..., 4, 5, 6], dtype=uint8)"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"test_labels"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Our workflow will be as follow: first we will present our neural network with the training data, `train_images` and `train_labels`. The \n",
"network will then learn to associate images and labels. Finally, we will ask the network to produce predictions for `test_images`, and we \n",
"will verify if these predictions match the labels from `test_labels`.\n",
"\n",
"Let's build our network -- again, remember that you aren't supposed to understand everything about this example just yet."
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"from keras import models\n",
"from keras import layers\n",
"\n",
"network = models.Sequential()\n",
"network.add(layers.Dense(512, activation='relu', input_shape=(28 * 28,)))\n",
"network.add(layers.Dense(10, activation='softmax'))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"The core building block of neural networks is the \"layer\", a data-processing module which you can conceive as a \"filter\" for data. Some \n",
"data comes in, and comes out in a more useful form. Precisely, layers extract _representations_ out of the data fed into them -- hopefully \n",
"representations that are more meaningful for the problem at hand. Most of deep learning really consists of chaining together simple layers \n",
"which will implement a form of progressive \"data distillation\". A deep learning model is like a sieve for data processing, made of a \n",
"succession of increasingly refined data filters -- the \"layers\".\n",
"\n",
"Here our network consists of a sequence of two `Dense` layers, which are densely-connected (also called \"fully-connected\") neural layers. \n",
"The second (and last) layer is a 10-way \"softmax\" layer, which means it will return an array of 10 probability scores (summing to 1). Each \n",
"score will be the probability that the current digit image belongs to one of our 10 digit classes.\n",
"\n",
"To make our network ready for training, we need to pick three more things, as part of \"compilation\" step:\n",
"\n",
"* A loss function: the is how the network will be able to measure how good a job it is doing on its training data, and thus how it will be \n",
"able to steer itself in the right direction.\n",
"* An optimizer: this is the mechanism through which the network will update itself based on the data it sees and its loss function.\n",
"* Metrics to monitor during training and testing. Here we will only care about accuracy (the fraction of the images that were correctly \n",
"classified).\n",
"\n",
"The exact purpose of the loss function and the optimizer will be made clear throughout the next two chapters."
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"network.compile(optimizer='rmsprop',\n",
" loss='categorical_crossentropy',\n",
" metrics=['accuracy'])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"Before training, we will preprocess our data by reshaping it into the shape that the network expects, and scaling it so that all values are in \n",
"the `[0, 1]` interval. Previously, our training images for instance were stored in an array of shape `(60000, 28, 28)` of type `uint8` with \n",
"values in the `[0, 255]` interval. We transform it into a `float32` array of shape `(60000, 28 * 28)` with values between 0 and 1."
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"train_images = train_images.reshape((60000, 28 * 28))\n",
"train_images = train_images.astype('float32') / 255\n",
"\n",
"test_images = test_images.reshape((10000, 28 * 28))\n",
"test_images = test_images.astype('float32') / 255"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We also need to categorically encode the labels, a step which we explain in chapter 3:"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"from keras.utils import to_categorical\n",
"\n",
"train_labels = to_categorical(train_labels)\n",
"test_labels = to_categorical(test_labels)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We are now ready to train our network, which in Keras is done via a call to the `fit` method of the network: \n",
"we \"fit\" the model to its training data."
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 1/5\n",
"60000/60000 [==============================] - 2s - loss: 0.2577 - acc: 0.9245 \n",
"Epoch 2/5\n",
"60000/60000 [==============================] - 1s - loss: 0.1042 - acc: 0.9690 \n",
"Epoch 3/5\n",
"60000/60000 [==============================] - 1s - loss: 0.0687 - acc: 0.9793 \n",
"Epoch 4/5\n",
"60000/60000 [==============================] - 1s - loss: 0.0508 - acc: 0.9848 \n",
"Epoch 5/5\n",
"60000/60000 [==============================] - 1s - loss: 0.0382 - acc: 0.9890 \n"
]
},
{
"data": {
"text/plain": [
"<keras.callbacks.History at 0x7fce5fed5fd0>"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"network.fit(train_images, train_labels, epochs=5, batch_size=128)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Two quantities are being displayed during training: the \"loss\" of the network over the training data, and the accuracy of the network over \n",
"the training data.\n",
"\n",
"We quickly reach an accuracy of 0.989 (i.e. 98.9%) on the training data. Now let's check that our model performs well on the test set too:"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
" 9536/10000 [===========================>..] - ETA: 0s"
]
}
],
"source": [
"test_loss, test_acc = network.evaluate(test_images, test_labels)"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"test_acc: 0.9777\n"
]
}
],
"source": [
"print('test_acc:', test_acc)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"Our test set accuracy turns out to be 97.8% -- that's quite a bit lower than the training set accuracy. \n",
"This gap between training accuracy and test accuracy is an example of \"overfitting\", \n",
"the fact that machine learning models tend to perform worse on new data than on their training data. \n",
"Overfitting will be a central topic in chapter 3.\n",
"\n",
"This concludes our very first example -- you just saw how we could build and a train a neural network to classify handwritten digits, in \n",
"less than 20 lines of Python code. In the next chapter, we will go in detail over every moving piece we just previewed, and clarify what is really \n",
"going on behind the scenes. You will learn about \"tensors\", the data-storing objects going into the network, about tensor operations, which \n",
"layers are made of, and about gradient descent, which allows our network to learn from its training examples."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.5.2"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
first_edition/3.5-classifying-movie-reviews.ipynb
+1051
View File
File diff suppressed because one or more lines are too long
first_edition/3.6-classifying-newswires.ipynb
+984
View File
File diff suppressed because one or more lines are too long
first_edition/3.7-predicting-house-prices.ipynb
+709
View File
File diff suppressed because one or more lines are too long
first_edition/4.4-overfitting-and-underfitting.ipynb
+980
View File
File diff suppressed because one or more lines are too long
first_edition/5.1-introduction-to-convnets.ipynb
+330
View File
@@ -0,0 +1,330 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Using TensorFlow backend.\n"
]
},
{
"data": {
"text/plain": [
"'2.0.8'"
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import keras\n",
"keras.__version__"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"# 5.1 - Introduction to convnets\n",
"\n",
"This notebook contains the code sample found in Chapter 5, Section 1 of [Deep Learning with Python](https://www.manning.com/books/deep-learning-with-python?a_aid=keras&a_bid=76564dff). Note that the original text features far more content, in particular further explanations and figures: in this notebook, you will only find source code and related comments.\n",
"\n",
"----\n",
"\n",
"First, let's take a practical look at a very simple convnet example. We will use our convnet to classify MNIST digits, a task that you've already been \n",
"through in Chapter 2, using a densely-connected network (our test accuracy then was 97.8%). Even though our convnet will be very basic, its \n",
"accuracy will still blow out of the water that of the densely-connected model from Chapter 2.\n",
"\n",
"The 6 lines of code below show you what a basic convnet looks like. It's a stack of `Conv2D` and `MaxPooling2D` layers. We'll see in a \n",
"minute what they do concretely.\n",
"Importantly, a convnet takes as input tensors of shape `(image_height, image_width, image_channels)` (not including the batch dimension). \n",
"In our case, we will configure our convnet to process inputs of size `(28, 28, 1)`, which is the format of MNIST images. We do this via \n",
"passing the argument `input_shape=(28, 28, 1)` to our first layer."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"from keras import layers\n",
"from keras import models\n",
"\n",
"model = models.Sequential()\n",
"model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(28, 28, 1)))\n",
"model.add(layers.MaxPooling2D((2, 2)))\n",
"model.add(layers.Conv2D(64, (3, 3), activation='relu'))\n",
"model.add(layers.MaxPooling2D((2, 2)))\n",
"model.add(layers.Conv2D(64, (3, 3), activation='relu'))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's display the architecture of our convnet so far:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"_________________________________________________________________\n",
"Layer (type) Output Shape Param # \n",
"=================================================================\n",
"conv2d_1 (Conv2D) (None, 26, 26, 32) 320 \n",
"_________________________________________________________________\n",
"max_pooling2d_1 (MaxPooling2 (None, 13, 13, 32) 0 \n",
"_________________________________________________________________\n",
"conv2d_2 (Conv2D) (None, 11, 11, 64) 18496 \n",
"_________________________________________________________________\n",
"max_pooling2d_2 (MaxPooling2 (None, 5, 5, 64) 0 \n",
"_________________________________________________________________\n",
"conv2d_3 (Conv2D) (None, 3, 3, 64) 36928 \n",
"=================================================================\n",
"Total params: 55,744\n",
"Trainable params: 55,744\n",
"Non-trainable params: 0\n",
"_________________________________________________________________\n"
]
}
],
"source": [
"model.summary()"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"You can see above that the output of every `Conv2D` and `MaxPooling2D` layer is a 3D tensor of shape `(height, width, channels)`. The width \n",
"and height dimensions tend to shrink as we go deeper in the network. The number of channels is controlled by the first argument passed to \n",
"the `Conv2D` layers (e.g. 32 or 64).\n",
"\n",
"The next step would be to feed our last output tensor (of shape `(3, 3, 64)`) into a densely-connected classifier network like those you are \n",
"already familiar with: a stack of `Dense` layers. These classifiers process vectors, which are 1D, whereas our current output is a 3D tensor. \n",
"So first, we will have to flatten our 3D outputs to 1D, and then add a few `Dense` layers on top:"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"model.add(layers.Flatten())\n",
"model.add(layers.Dense(64, activation='relu'))\n",
"model.add(layers.Dense(10, activation='softmax'))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We are going to do 10-way classification, so we use a final layer with 10 outputs and a softmax activation. Now here's what our network \n",
"looks like:"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"_________________________________________________________________\n",
"Layer (type) Output Shape Param # \n",
"=================================================================\n",
"conv2d_1 (Conv2D) (None, 26, 26, 32) 320 \n",
"_________________________________________________________________\n",
"max_pooling2d_1 (MaxPooling2 (None, 13, 13, 32) 0 \n",
"_________________________________________________________________\n",
"conv2d_2 (Conv2D) (None, 11, 11, 64) 18496 \n",
"_________________________________________________________________\n",
"max_pooling2d_2 (MaxPooling2 (None, 5, 5, 64) 0 \n",
"_________________________________________________________________\n",
"conv2d_3 (Conv2D) (None, 3, 3, 64) 36928 \n",
"_________________________________________________________________\n",
"flatten_1 (Flatten) (None, 576) 0 \n",
"_________________________________________________________________\n",
"dense_1 (Dense) (None, 64) 36928 \n",
"_________________________________________________________________\n",
"dense_2 (Dense) (None, 10) 650 \n",
"=================================================================\n",
"Total params: 93,322\n",
"Trainable params: 93,322\n",
"Non-trainable params: 0\n",
"_________________________________________________________________\n"
]
}
],
"source": [
"model.summary()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As you can see, our `(3, 3, 64)` outputs were flattened into vectors of shape `(576,)`, before going through two `Dense` layers.\n",
"\n",
"Now, let's train our convnet on the MNIST digits. We will reuse a lot of the code we have already covered in the MNIST example from Chapter \n",
"2."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"from keras.datasets import mnist\n",
"from keras.utils import to_categorical\n",
"\n",
"(train_images, train_labels), (test_images, test_labels) = mnist.load_data()\n",
"\n",
"train_images = train_images.reshape((60000, 28, 28, 1))\n",
"train_images = train_images.astype('float32') / 255\n",
"\n",
"test_images = test_images.reshape((10000, 28, 28, 1))\n",
"test_images = test_images.astype('float32') / 255\n",
"\n",
"train_labels = to_categorical(train_labels)\n",
"test_labels = to_categorical(test_labels)"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch 1/5\n",
"60000/60000 [==============================] - 8s - loss: 0.1766 - acc: 0.9440 \n",
"Epoch 2/5\n",
"60000/60000 [==============================] - 7s - loss: 0.0462 - acc: 0.9855 \n",
"Epoch 3/5\n",
"60000/60000 [==============================] - 7s - loss: 0.0322 - acc: 0.9902 \n",
"Epoch 4/5\n",
"60000/60000 [==============================] - 7s - loss: 0.0241 - acc: 0.9926 \n",
"Epoch 5/5\n",
"60000/60000 [==============================] - 7s - loss: 0.0187 - acc: 0.9943 \n"
]
},
{
"data": {
"text/plain": [
"<keras.callbacks.History at 0x7fbd9c4cd828>"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"model.compile(optimizer='rmsprop',\n",
" loss='categorical_crossentropy',\n",
" metrics=['accuracy'])\n",
"model.fit(train_images, train_labels, epochs=5, batch_size=64)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's evaluate the model on the test data:"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
" 9536/10000 [===========================>..] - ETA: 0s"
]
}
],
"source": [
"test_loss, test_acc = model.evaluate(test_images, test_labels)"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.99129999999999996"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"test_acc"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"While our densely-connected network from Chapter 2 had a test accuracy of 97.8%, our basic convnet has a test accuracy of 99.3%: we \n",
"decreased our error rate by 68% (relative). Not bad! "
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.5.2"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
first_edition/5.2-using-convnets-with-small-datasets.ipynb
+1275
View File
File diff suppressed because one or more lines are too long
first_edition/5.3-using-a-pretrained-convnet.ipynb
+1417
View File
File diff suppressed because one or more lines are too long
first_edition/5.4-visualizing-what-convnets-learn.ipynb
+1152
View File
File diff suppressed because one or more lines are too long
first_edition/6.1-one-hot-encoding-of-words-or-characters.ipynb
+243
View File
@@ -0,0 +1,243 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Using TensorFlow backend.\n"
]
},
{
"data": {
"text/plain": [
"'2.0.8'"
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import keras\n",
"keras.__version__"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# One-hot encoding of words or characters\n",
"\n",
"This notebook contains the first code sample found in Chapter 6, Section 1 of [Deep Learning with Python](https://www.manning.com/books/deep-learning-with-python?a_aid=keras&a_bid=76564dff). Note that the original text features far more content, in particular further explanations and figures: in this notebook, you will only find source code and related comments.\n",
"\n",
"----\n",
"\n",
"One-hot encoding is the most common, most basic way to turn a token into a vector. You already saw it in action in our initial IMDB and \n",
"Reuters examples from chapter 3 (done with words, in our case). It consists in associating a unique integer index to every word, then \n",
"turning this integer index i into a binary vector of size N, the size of the vocabulary, that would be all-zeros except for the i-th \n",
"entry, which would be 1.\n",
"\n",
"Of course, one-hot encoding can be done at the character level as well. To unambiguously drive home what one-hot encoding is and how to \n",
"implement it, here are two toy examples of one-hot encoding: one for words, the other for characters.\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Word level one-hot encoding (toy example):"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"\n",
"# This is our initial data; one entry per \"sample\"\n",
"# (in this toy example, a \"sample\" is just a sentence, but\n",
"# it could be an entire document).\n",
"samples = ['The cat sat on the mat.', 'The dog ate my homework.']\n",
"\n",
"# First, build an index of all tokens in the data.\n",
"token_index = {}\n",
"for sample in samples:\n",
" # We simply tokenize the samples via the `split` method.\n",
" # in real life, we would also strip punctuation and special characters\n",
" # from the samples.\n",
" for word in sample.split():\n",
" if word not in token_index:\n",
" # Assign a unique index to each unique word\n",
" token_index[word] = len(token_index) + 1\n",
" # Note that we don't attribute index 0 to anything.\n",
"\n",
"# Next, we vectorize our samples.\n",
"# We will only consider the first `max_length` words in each sample.\n",
"max_length = 10\n",
"\n",
"# This is where we store our results:\n",
"results = np.zeros((len(samples), max_length, max(token_index.values()) + 1))\n",
"for i, sample in enumerate(samples):\n",
" for j, word in list(enumerate(sample.split()))[:max_length]:\n",
" index = token_index.get(word)\n",
" results[i, j, index] = 1."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Character level one-hot encoding (toy example)"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"import string\n",
"\n",
"samples = ['The cat sat on the mat.', 'The dog ate my homework.']\n",
"characters = string.printable # All printable ASCII characters.\n",
"token_index = dict(zip(characters, range(1, len(characters) + 1)))\n",
"\n",
"max_length = 50\n",
"results = np.zeros((len(samples), max_length, max(token_index.values()) + 1))\n",
"for i, sample in enumerate(samples):\n",
" for j, character in enumerate(sample[:max_length]):\n",
" index = token_index.get(character)\n",
" results[i, j, index] = 1."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Note that Keras has built-in utilities for doing one-hot encoding text at the word level or character level, starting from raw text data. \n",
"This is what you should actually be using, as it will take care of a number of important features, such as stripping special characters \n",
"from strings, or only taking into the top N most common words in your dataset (a common restriction to avoid dealing with very large input \n",
"vector spaces)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Using Keras for word-level one-hot encoding:"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Found 9 unique tokens.\n"
]
}
],
"source": [
"from keras.preprocessing.text import Tokenizer\n",
"\n",
"samples = ['The cat sat on the mat.', 'The dog ate my homework.']\n",
"\n",
"# We create a tokenizer, configured to only take\n",
"# into account the top-1000 most common words\n",
"tokenizer = Tokenizer(num_words=1000)\n",
"# This builds the word index\n",
"tokenizer.fit_on_texts(samples)\n",
"\n",
"# This turns strings into lists of integer indices.\n",
"sequences = tokenizer.texts_to_sequences(samples)\n",
"\n",
"# You could also directly get the one-hot binary representations.\n",
"# Note that other vectorization modes than one-hot encoding are supported!\n",
"one_hot_results = tokenizer.texts_to_matrix(samples, mode='binary')\n",
"\n",
"# This is how you can recover the word index that was computed\n",
"word_index = tokenizer.word_index\n",
"print('Found %s unique tokens.' % len(word_index))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"A variant of one-hot encoding is the so-called \"one-hot hashing trick\", which can be used when the number of unique tokens in your \n",
"vocabulary is too large to handle explicitly. Instead of explicitly assigning an index to each word and keeping a reference of these \n",
"indices in a dictionary, one may hash words into vectors of fixed size. This is typically done with a very lightweight hashing function. \n",
"The main advantage of this method is that it does away with maintaining an explicit word index, which \n",
"saves memory and allows online encoding of the data (starting to generate token vectors right away, before having seen all of the available \n",
"data). The one drawback of this method is that it is susceptible to \"hash collisions\": two different words may end up with the same hash, \n",
"and subsequently any machine learning model looking at these hashes won't be able to tell the difference between these words. The likelihood \n",
"of hash collisions decreases when the dimensionality of the hashing space is much larger than the total number of unique tokens being hashed."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Word-level one-hot encoding with hashing trick (toy example):"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
"samples = ['The cat sat on the mat.', 'The dog ate my homework.']\n",
"\n",
"# We will store our words as vectors of size 1000.\n",
"# Note that if you have close to 1000 words (or more)\n",
"# you will start seeing many hash collisions, which\n",
"# will decrease the accuracy of this encoding method.\n",
"dimensionality = 1000\n",
"max_length = 10\n",
"\n",
"results = np.zeros((len(samples), max_length, dimensionality))\n",
"for i, sample in enumerate(samples):\n",
" for j, word in list(enumerate(sample.split()))[:max_length]:\n",
" # Hash the word into a \"random\" integer index\n",
" # that is between 0 and 1000\n",
" index = abs(hash(word)) % dimensionality\n",
" results[i, j, index] = 1."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.5.2"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
first_edition/6.1-using-word-embeddings.ipynb
+946
View File
File diff suppressed because one or more lines are too long
first_edition/6.2-understanding-recurrent-neural-networks.ipynb
+487
View File
File diff suppressed because one or more lines are too long
first_edition/6.3-advanced-usage-of-recurrent-neural-networks.ipynb
+1676
View File
File diff suppressed because one or more lines are too long
first_edition/6.4-sequence-processing-with-convnets.ipynb
+720
View File
File diff suppressed because one or more lines are too long
first_edition/8.1-text-generation-with-lstm.ipynb
+2345
View File
File diff suppressed because it is too large Load Diff
first_edition/8.2-deep-dream.ipynb
+437
View File
File diff suppressed because one or more lines are too long
first_edition/8.3-neural-style-transfer.ipynb
+723
View File
File diff suppressed because one or more lines are too long
first_edition/8.4-generating-images-with-vaes.ipynb
+494
View File
File diff suppressed because one or more lines are too long
first_edition/8.5-introduction-to-gans.ipynb
+783
View File
File diff suppressed because one or more lines are too long