wehub-resource-sync
721 lines
92 KiB
Plaintext
721 lines
92 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Using TensorFlow backend.\n"
|
|
]
|
|
},
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"'2.0.8'"
|
|
]
|
|
},
|
|
"execution_count": 1,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"import keras\n",
|
|
"keras.__version__"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Sequence processing with convnets\n",
|
|
"\n",
|
|
"This notebook contains the code samples found in Chapter 6, Section 4 of [Deep Learning with Python](https://www.manning.com/books/deep-learning-with-python?a_aid=keras&a_bid=76564dff). Note that the original text features far more content, in particular further explanations and figures: in this notebook, you will only find source code and related comments.\n",
|
|
"\n",
|
|
"\n",
|
|
"## Implementing a 1D convnet\n",
|
|
"\n",
|
|
"In Keras, you would use a 1D convnet via the `Conv1D` layer, which has a very similar interface to `Conv2D`. It takes as input 3D tensors \n",
|
|
"with shape `(samples, time, features)` and also returns similarly-shaped 3D tensors. The convolution window is a 1D window on the temporal \n",
|
|
"axis, axis 1 in the input tensor.\n",
|
|
"\n",
|
|
"Let's build a simple 2-layer 1D convnet and apply it to the IMDB sentiment classification task that you are already familiar with.\n",
|
|
"\n",
|
|
"As a reminder, this is the code for obtaining and preprocessing the data:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Loading data...\n",
|
|
"25000 train sequences\n",
|
|
"25000 test sequences\n",
|
|
"Pad sequences (samples x time)\n",
|
|
"x_train shape: (25000, 500)\n",
|
|
"x_test shape: (25000, 500)\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"from keras.datasets import imdb\n",
|
|
"from keras.preprocessing import sequence\n",
|
|
"\n",
|
|
"max_features = 10000 # number of words to consider as features\n",
|
|
"max_len = 500 # cut texts after this number of words (among top max_features most common words)\n",
|
|
"\n",
|
|
"print('Loading data...')\n",
|
|
"(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=max_features)\n",
|
|
"print(len(x_train), 'train sequences')\n",
|
|
"print(len(x_test), 'test sequences')\n",
|
|
"\n",
|
|
"print('Pad sequences (samples x time)')\n",
|
|
"x_train = sequence.pad_sequences(x_train, maxlen=max_len)\n",
|
|
"x_test = sequence.pad_sequences(x_test, maxlen=max_len)\n",
|
|
"print('x_train shape:', x_train.shape)\n",
|
|
"print('x_test shape:', x_test.shape)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"\n",
|
|
"1D convnets are structured in the same way as their 2D counter-parts that you have used in Chapter 5: they consist of a stack of `Conv1D` \n",
|
|
"and `MaxPooling1D` layers, eventually ending in either a global pooling layer or a `Flatten` layer, turning the 3D outputs into 2D outputs, \n",
|
|
"allowing to add one or more `Dense` layers to the model, for classification or regression.\n",
|
|
"\n",
|
|
"One difference, though, is the fact that we can afford to use larger convolution windows with 1D convnets. Indeed, with a 2D convolution \n",
|
|
"layer, a 3x3 convolution window contains 3*3 = 9 feature vectors, but with a 1D convolution layer, a convolution window of size 3 would \n",
|
|
"only contain 3 feature vectors. We can thus easily afford 1D convolution windows of size 7 or 9.\n",
|
|
"\n",
|
|
"This is our example 1D convnet for the IMDB dataset:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"_________________________________________________________________\n",
|
|
"Layer (type) Output Shape Param # \n",
|
|
"=================================================================\n",
|
|
"embedding_1 (Embedding) (None, 500, 128) 1280000 \n",
|
|
"_________________________________________________________________\n",
|
|
"conv1d_1 (Conv1D) (None, 494, 32) 28704 \n",
|
|
"_________________________________________________________________\n",
|
|
"max_pooling1d_1 (MaxPooling1 (None, 98, 32) 0 \n",
|
|
"_________________________________________________________________\n",
|
|
"conv1d_2 (Conv1D) (None, 92, 32) 7200 \n",
|
|
"_________________________________________________________________\n",
|
|
"global_max_pooling1d_1 (Glob (None, 32) 0 \n",
|
|
"_________________________________________________________________\n",
|
|
"dense_1 (Dense) (None, 1) 33 \n",
|
|
"=================================================================\n",
|
|
"Total params: 1,315,937\n",
|
|
"Trainable params: 1,315,937\n",
|
|
"Non-trainable params: 0\n",
|
|
"_________________________________________________________________\n",
|
|
"Train on 20000 samples, validate on 5000 samples\n",
|
|
"Epoch 1/10\n",
|
|
"20000/20000 [==============================] - 4s - loss: 0.7713 - acc: 0.5287 - val_loss: 0.6818 - val_acc: 0.5970\n",
|
|
"Epoch 2/10\n",
|
|
"20000/20000 [==============================] - 3s - loss: 0.6631 - acc: 0.6775 - val_loss: 0.6582 - val_acc: 0.6646\n",
|
|
"Epoch 3/10\n",
|
|
"20000/20000 [==============================] - 3s - loss: 0.6142 - acc: 0.7580 - val_loss: 0.5987 - val_acc: 0.7118\n",
|
|
"Epoch 4/10\n",
|
|
"20000/20000 [==============================] - 3s - loss: 0.5156 - acc: 0.8124 - val_loss: 0.4936 - val_acc: 0.7736\n",
|
|
"Epoch 5/10\n",
|
|
"20000/20000 [==============================] - 3s - loss: 0.4029 - acc: 0.8469 - val_loss: 0.4123 - val_acc: 0.8358\n",
|
|
"Epoch 6/10\n",
|
|
"20000/20000 [==============================] - 3s - loss: 0.3455 - acc: 0.8653 - val_loss: 0.4040 - val_acc: 0.8382\n",
|
|
"Epoch 7/10\n",
|
|
"20000/20000 [==============================] - 3s - loss: 0.3078 - acc: 0.8634 - val_loss: 0.4059 - val_acc: 0.8240\n",
|
|
"Epoch 8/10\n",
|
|
"20000/20000 [==============================] - 3s - loss: 0.2812 - acc: 0.8535 - val_loss: 0.4147 - val_acc: 0.8098\n",
|
|
"Epoch 9/10\n",
|
|
"20000/20000 [==============================] - 3s - loss: 0.2554 - acc: 0.8334 - val_loss: 0.4296 - val_acc: 0.7878\n",
|
|
"Epoch 10/10\n",
|
|
"20000/20000 [==============================] - 3s - loss: 0.2356 - acc: 0.8052 - val_loss: 0.4296 - val_acc: 0.7600\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"from keras.models import Sequential\n",
|
|
"from keras import layers\n",
|
|
"from keras.optimizers import RMSprop\n",
|
|
"\n",
|
|
"model = Sequential()\n",
|
|
"model.add(layers.Embedding(max_features, 128, input_length=max_len))\n",
|
|
"model.add(layers.Conv1D(32, 7, activation='relu'))\n",
|
|
"model.add(layers.MaxPooling1D(5))\n",
|
|
"model.add(layers.Conv1D(32, 7, activation='relu'))\n",
|
|
"model.add(layers.GlobalMaxPooling1D())\n",
|
|
"model.add(layers.Dense(1))\n",
|
|
"\n",
|
|
"model.summary()\n",
|
|
"\n",
|
|
"model.compile(optimizer=RMSprop(lr=1e-4),\n",
|
|
" loss='binary_crossentropy',\n",
|
|
" metrics=['acc'])\n",
|
|
"history = model.fit(x_train, y_train,\n",
|
|
" epochs=10,\n",
|
|
" batch_size=128,\n",
|
|
" validation_split=0.2)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Here are our training and validation results: validation accuracy is somewhat lower than that of the LSTM we used two sections ago, but \n",
|
|
"runtime is faster, both on CPU and GPU (albeit the exact speedup will vary greatly depending on your exact configuration). At that point, \n",
|
|
"we could re-train this model for the right number of epochs (8), and run it on the test set. This is a convincing demonstration that a 1D \n",
|
|
"convnet can offer a fast, cheap alternative to a recurrent network on a word-level sentiment classification task."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEICAYAAACzliQjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl4FFXW+PHvIYAhCLI6IAiJirIoa0QZQGAQxQ1cEEFw\nBBcUBdRx5hUEFVEcRx1f1JdRo4OjEkB+MiKoiCIouCAJQkSCILIZQIyIbIlC4Pz+uJWkE7I00El1\nus/nefrprqpb1acr6VPVt27dK6qKMcaY6FDJ7wCMMcaUH0v6xhgTRSzpG2NMFLGkb4wxUcSSvjHG\nRBFL+sYYE0Us6UchEYkRkX0i0iSUZf0kImeISMjbH4vIhSKyKWB6rYh0DabsMbzXyyJy/7Gub0ww\nKvsdgCmdiOwLmIwDfgcOedO3qWry0WxPVQ8BJ4a6bDRQ1bNCsR0RuQUYrKrdA7Z9Syi2bUxJLOlX\nAKqal3S9M8lbVHVBceVFpLKq5pRHbMaUxv4fw4tV70QAEXlURN4QkekishcYLCKdRGSpiPwqIttF\n5FkRqeKVrywiKiLx3vRUb/k8EdkrIl+ISMLRlvWWXyIi60Rkt4g8JyKficiQYuIOJsbbRGS9iOwS\nkWcD1o0Rkf8VkZ0isgHoXcL+GSsiMwrNmywiT3uvbxGRNd7n+d47Cy9uWxki0t17HScir3uxrQY6\nFCo7TkQ2eNtdLSJ9vPnnAP8HdPWqzn4O2LfjA9a/3fvsO0Vktog0DGbfHM1+zo1HRBaIyC8i8qOI\n/E/A+zzg7ZM9IpIqIqcUVZUmIp/m/p29/bnYe59fgHEi0kxEFnnv8bO3304KWL+p9xkzveXPiEis\nF3OLgHINRSRLROoW93lNKVTVHhXoAWwCLiw071HgAHAF7kBeDTgXOA/3a+40YB0wwitfGVAg3pue\nCvwMJAJVgDeAqcdQ9mRgL9DXW/YX4CAwpJjPEkyMbwMnAfHAL7mfHRgBrAYaA3WBxe7fucj3OQ3Y\nB1QP2PZPQKI3fYVXRoA/AdlAa2/ZhcCmgG1lAN29108BHwO1gaZAeqGy/YGG3t/kei+GP3jLbgE+\nLhTnVGC89/oiL8a2QCzwL2BhMPvmKPfzScAO4C7gBKAm0NFbNgZIA5p5n6EtUAc4o/C+Bj7N/Tt7\nny0HGA7E4P4fzwR6AlW9/5PPgKcCPs833v6s7pXv7C1LAiYGvM+9wFt+fw8r8sP3AOxxlH+w4pP+\nwlLW+yvw/7zXRSXyFwLK9gG+OYayNwFLApYJsJ1ikn6QMZ4fsPy/wF+914tx1Vy5yy4tnIgKbXsp\ncL33+hJgbQll3wHu9F6XlPS3BP4tgDsCyxax3W+Ay7zXpSX9V4HHApbVxF3HaVzavjnK/XwDkFJM\nue9z4y00P5ikv6GUGPrlvi/QFfgRiCmiXGdgIyDe9Erg6lB/r6LpYdU7keOHwAkRaS4i73o/1/cA\nE4B6Jaz/Y8DrLEq+eFtc2VMC41D3Lc0obiNBxhjUewGbS4gXYBow0Ht9vTedG8flIvKlV/XwK+4s\nu6R9lathSTGIyBARSfOqKH4Fmge5XXCfL297qroH2AU0CigT1N+slP18Ki65F6WkZaUp/P/YQERm\nishWL4b/FIphk7pGAwWo6me4Xw1dRORsoAnw7jHGZLA6/UhSuLnii7gzyzNUtSbwIO7Muyxtx52J\nAiAiQsEkVdjxxLgdlyxyldakdCZwoYg0wlU/TfNirAa8CfwdV/VSC/ggyDh+LC4GETkNeB5XxVHX\n2+63AdstrXnpNlyVUe72auCqkbYGEVdhJe3nH4DTi1mvuGX7vZjiAuY1KFSm8Of7B67V2TleDEMK\nxdBURGKKieM1YDDuV8lMVf29mHImCJb0I1cNYDew37sQdls5vOc7QHsRuUJEKuPqieuXUYwzgbtF\npJF3Ue++kgqr6o+4Koj/4Kp2vvMWnYCrZ84EDonI5bi652BjuF9Eaom7j2FEwLITcYkvE3f8uxV3\npp9rB9A48IJqIdOBm0WktYicgDsoLVHVYn85laCk/TwHaCIiI0TkBBGpKSIdvWUvA4+KyOnitBWR\nOriD3Y+4BgMxIjKMgANUCTHsB3aLyKm4KqZcXwA7gcfEXRyvJiKdA5a/jqsOuh53ADDHwZJ+5LoX\nuBF3YfVF3AXXMqWqO4DrgKdxX+LTgRW4M7xQx/g88BGwCkjBna2XZhqujj6vakdVfwXuAd7CXQzt\nhzt4BeMh3C+OTcA8AhKSqn4NPAcs88qcBXwZsO6HwHfADhEJrKbJXf99XDXMW976TYBBQcZVWLH7\nWVV3A72Aa3AHonVAN2/xk8Bs3H7eg7uoGutV290K3I+7qH9Goc9WlIeAjriDzxxgVkAMOcDlQAvc\nWf8W3N8hd/km3N/5d1X9/Cg/uykk9+KIMSHn/VzfBvRT1SV+x2MqLhF5DXdxeLzfsVR0dnOWCSkR\n6Y1rKZONa/J3EHe2a8wx8a6P9AXO8TuWSGDVOybUugAbcHXZFwNX2YU3c6xE5O+4ewUeU9UtfscT\nCax6xxhjooid6RtjTBQJuzr9evXqaXx8vN9hGGNMhbJ8+fKfVbWkJtJAGCb9+Ph4UlNT/Q7DGGMq\nFBEp7a50wKp3jDEmqljSN8aYKGJJ3xhjokjY1ekX5eDBg2RkZPDbb7/5HYopQWxsLI0bN6ZKleK6\nkzHG+K1CJP2MjAxq1KhBfHw8ruNGE25UlZ07d5KRkUFCQkLpKxhjfFEhqnd+++036tatawk/jIkI\ndevWtV9jJUhOhvh4qFTJPScf1XD2xoRGhUj6gCX8CsD+RsVLToZhw2DzZlB1z8OG+ZP47eAT3SpM\n0jemIhs7FrKyCs7LynLzy1M4HXyMPyzpB2Hnzp20bduWtm3b0qBBAxo1apQ3feDAgaC2MXToUNau\nXVtimcmTJ5Ns376ItKWYrsKKm19WwuXgY/xTIS7kHq3kZPdPvGULNGkCEyfCoGMdfgKoW7cuK1eu\nBGD8+PGceOKJ/PWvfy1QJm/Q4UpFH0dfeeWVUt/nzjvvPPYgTVhr0sSdVRc1vzyFy8HH+CfizvTL\n8+fr+vXradmyJYMGDaJVq1Zs376dYcOGkZiYSKtWrZgwYUJe2S5durBy5UpycnKoVasWo0ePpk2b\nNnTq1ImffvoJgHHjxjFp0qS88qNHj6Zjx46cddZZfP65GzBo//79XHPNNbRs2ZJ+/fqRmJiYd0AK\n9NBDD3Huuedy9tlnc/vtt5Pbm+q6dev405/+RJs2bWjfvj2bNm0C4LHHHuOcc86hTZs2jLXTvpCb\nOBHi4grOi4tz88tTcQeZ8j74GP9EXNIv75+v3377Lffccw/p6ek0atSIxx9/nNTUVNLS0vjwww9J\nT08/Yp3du3fTrVs30tLS6NSpE1OmTCly26rKsmXLePLJJ/MOIM899xwNGjQgPT2dBx54gBUrVhS5\n7l133UVKSgqrVq1i9+7dvP/++wAMHDiQe+65h7S0ND7//HNOPvlk5s6dy7x581i2bBlpaWnce++9\nIdo7JtegQZCUBE2bgoh7Tko6vl+gxyJcDj7GPxGX9Mv75+vpp59OYmJi3vT06dNp37497du3Z82a\nNUUm/WrVqnHJJZcA0KFDh7yz7cKuvvrqI8p8+umnDBgwAIA2bdrQqlWrItf96KOP6NixI23atOGT\nTz5h9erV7Nq1i59//pkrrrgCcDdTxcXFsWDBAm666SaqVasGQJ06dY5+R5hSDRoEmzbB4cPuubwT\nfm4M4XDwsRZE/om4Ov3yrjutXr163uvvvvuOZ555hmXLllGrVi0GDx5cZLv1qlWr5r2OiYkhJyen\nyG2fcMIJpZYpSlZWFiNGjOCrr76iUaNGjBs3ztrPmzyDBvlzwMmVWwWb+4s8two2NzZTtoI60xeR\n3iKyVkTWi8joIpY3EZFFIrJCRL4WkUu9+fEiki0iK73HC6H+AIX5+fN1z5491KhRg5o1a7J9+3bm\nz58f8vfo3LkzM2fOBGDVqlVF/pLIzs6mUqVK1KtXj7179zJr1iwAateuTf369Zk7dy7gbnrLysqi\nV69eTJkyhezsbAB++eWXkMdtTC5rQeSvUs/0RSQGmAz0AjKAFBGZo6qB2WYcMFNVnxeRlsB7QLy3\n7HtVbRvasIuXe6YQytY7wWrfvj0tW7akefPmNG3alM6dO4f8PUaOHMmf//xnWrZsmfc46aSTCpSp\nW7cuN954Iy1btqRhw4acd955ecuSk5O57bbbGDt2LFWrVmXWrFlcfvnlpKWlkZiYSJUqVbjiiit4\n5JFHQh67MWAtiHyX29SwuAfQCZgfMD0GGFOozIvAfQHlP/dexwPflPYegY8OHTpoYenp6UfMi1YH\nDx7U7OxsVVVdt26dxsfH68GDB32OKl84/q2mTlVt2lRVxD1Pnep3RNGtaVNV17au4KNpU78jq9iA\nVA0ixwZTp98I+CFgOgM4r1CZ8cAHIjISqA5cGLAsQURWAHuAcaq6pPAbiMgwYBhAE2s7VqJ9+/bR\ns2dPcnJyUFVefPFFKleOuEszIWP1x+Fn4sSCfxOwFkTlKVTZYiDwH1X9p4h0Al4XkbOB7UATVd0p\nIh2A2SLSSlX3BK6sqklAEkBiYqKGKKaIVKtWLZYvX+53GBVGSfXHlvT94WcVrAku6W8FTg2YbuzN\nC3Qz0BtAVb8QkVignqr+BPzuzV8uIt8DZwI2CK4pF1Z/HJ78bkEUzYJpvZMCNBORBBGpCgwA5hQq\nswXoCSAiLYBYIFNE6nsXghGR04BmwIZQBW9MaewOVGMKKjXpq2oOMAKYD6zBtdJZLSITRKSPV+xe\n4FYRSQOmA0O8CwsXAF+LyErgTeB2VbX2gKbc2B2oxhQUVJ2+qr6Ha4YZOO/BgNfpwBHtE1V1FjDr\nOGM05phZ/bExBVmzjyD06NGD0aNHc/HFF+fNmzRpEmvXruX5558vdr0TTzyRffv2sW3bNkaNGsWb\nb755RJnu3bvz1FNPFejKobBJkyYxbNgw4rxT1ksvvZRp06ZRq1at4/hU0aMs6o8PH4bsbHdReP9+\n9wjmdXY2NGsG3bpB69auGwJjypMl/SAMHDiQGTNmFEj6M2bM4Iknnghq/VNOOaXIhB+sSZMmMXjw\n4Lyk/95775Wyhilszx5Yu/bIJBxssi78unCLoGBUqwZVq8Lu3W66Vi3o2tUdALp1g7ZtwVrflq9Q\nd8NeEdi/WBD69evHuHHjOHDgAFWrVmXTpk1s27aNrl27sm/fPvr27cuuXbs4ePAgjz76KH379i2w\n/qZNm7j88sv55ptvyM7OZujQoaSlpdG8efO8rg8Ahg8fTkpKCtnZ2fTr14+HH36YZ599lm3bttGj\nRw/q1avHokWLiI+PJzU1lXr16vH000/n9dJ5yy23cPfdd7Np0yYuueQSunTpwueff06jRo14++23\n8zpUyzV37lweffRRDhw4QN26dUlOTuYPf/gD+/btY+TIkaSmpiIiPPTQQ1xzzTW8//773H///Rw6\ndIh69erx0Ucflf3OD4GNG6FLF9i2reRy1aq5+v7q1d0j93X9+kXPP5rXcXH5Z/VbtsAnn+Q/vF4x\nqFGj4EGgfXuoUqVs9000i9Z7OCpc0r/7biii+/jj0rYteN3YF6lOnTp07NiRefPm0bdvX2bMmEH/\n/v0REWJjY3nrrbeoWbMmP//8M+effz59+vQpdrzY559/nri4ONasWcPXX39N+/bt85ZNnDiROnXq\ncOjQIXr27MnXX3/NqFGjePrpp1m0aBH16tUrsK3ly5fzyiuv8OWXX6KqnHfeeXTr1o3atWvz3Xff\nMX36dF566SX69+/PrFmzGDx4cIH1u3TpwtKlSxERXn75ZZ544gn++c9/8sgjj3DSSSexatUqAHbt\n2kVmZia33norixcvJiEhocL0z/Pjj9Crl6tWeeMNl8BLS8plrUkTuOEG9wDYuhUWL84/COT+kKte\nHTp3zj8InHuu+6VgQiNa7+GocEnfL7lVPLlJ/9///jfgurG4//77Wbx4MZUqVWLr1q3s2LGDBg0a\nFLmdxYsXM2rUKABat25N69at85bNnDmTpKQkcnJy2L59O+np6QWWF/bpp59y1VVX5fX0efXVV7Nk\nyRL69OlDQkICbdu6Lo+K6745IyOD6667ju3bt3PgwAESEhIAWLBgATNmzMgrV7t2bebOncsFF1yQ\nV6YidL/866/Quzds3w4LFkCnTn5HVLRGjWDgQPcAd6AKPAjkdkRWrRr88Y/5B4GOHSE21r+4K7po\nvYejwiX9ks7Iy1Lfvn255557+Oqrr8jKyqJDhw6A68AsMzOT5cuXU6VKFeLj44+pG+ONGzfy1FNP\nkZKSQu3atRkyZMhxdYec2y0zuK6ZA6uRco0cOZK//OUv9OnTh48//pjx48cf8/uFm6wsuPxySE+H\nd94J34RflAYNoH9/9wDIzIQlS/IPAg895HqrOeEEOP/8/INAp07uwGCCEy5DWJY3azsQpBNPPJEe\nPXpw0003MTD3lAw3CtbJJ59MlSpVWLRoEZuL+i8KcMEFFzBt2jQAvvnmG77++mvAdctcvXp1Tjrp\nJHbs2MG8efPy1qlRowZ79+49Yltdu3Zl9uzZZGVlsX//ft566y26du0a9GfavXs3jRo1AuDVV1/N\nm9+rVy8mT56cN71r1y7OP/98Fi9ezMaNG4Hw7n754EG49lr4/HOYOhUuusjviI5P/fpw9dXwzDOu\navPnn2H2bLjjDti7Fx59FHr2zL8wPG4cfPihu+hsihet93BY0j8KAwcOJC0trUDSHzRoEKmpqZxz\nzjm89tprNG/evMRtDB8+nH379tGiRQsefPDBvF8Mbdq0oV27djRv3pzrr7++QLfMw4YNo3fv3vTo\n0aPAttq3b8+QIUPo2LEj5513Hrfccgvt2rUL+vOMHz+ea6+9lg4dOhS4XjBu3Dh27drF2WefTZs2\nbVi0aBH169cnKSmJq6++mjZt2nDdddcF/T7l6fBhGDLE1Yu/8EL+2XIkqVMH+vaFp5+G5cvhl1/c\nr5m77oIDB+Dxx92BrlYtd/Y/ejTMm+daMJl84TKKWHkT1fDq3ywxMVFTUwt2zbNmzRpatGjhU0Tm\naPj5t1KFkSNh8mR47DEYM8aXMHy3d6/7lfPxx646KCUFcnLcheoOHfKrg7p0cQcGExlEZLmqFn/D\nj6fC1ekbU5zx413Cv/ded3YbrWrUgIsvdg9w1TxffJF/TeDZZ+Gpp9zZbdu2+QeBCy5wvyJMZLOk\nbyLCs8/ChAkwdCg8+aRLaMapXh0uvNA9wDVfXbo0/yDwwguugYQInHNOwYNA/fr+xm5Cr8IkfVUt\ntu27CQ9+VRW+/rqrz77ySlcna/8mJatWDXr0cA+A33+HZcvyDwL//jc895xb1rJl/kGgWzfXsshU\nbBWiTn/jxo3UqFGDunXrWuIPU6rKzp072bt3b15b/vIwdy5cdZVLSO++a+3WQ+HAAXeBOPeawGef\nwb59btlZZxU8CHiNv0wYCLZOv0Ik/YMHD5KRkXFc7dZN2YuNjaVx48ZU8foOKOt+TRYvdvXWZ58N\nCxe6umwTejk58NVX+b8ElizJbwl0+uku+Xfv7p4jvY17OIuopG8qnsL9moBrAx2qJnErVrhEc8op\nLgkV6qHClKFDhyAtLf+XwJIlsGuXWxYfX/CXQEKCVbeVF0v6xlfx8UXf7di0KRTRI8RRWbfONTes\nVs1VPTRufHzbM8fn8GFYtSr/l8Dixe4GMnB/m9xfAd26wRln2EGgrFjSN76qVMm1my9MxCWJY5WR\n4Tohy86GTz+FM8889m2ZsnH4sOv+IrAn0Z9+cssaNiz4S6B5czsIhIq10ze+Kot+TX7+2d1pumuX\nq1qwhB+eKlVy11nOPhvuvNMd/NeuLXgQyO3Pr2FDuP56uOkm11LIlD3rhsGUiVD3a7J3L1x6qesb\nf+5c19e8qRhE3Bn9bbfBtGnu19p338FLL8F557k+hVq1cp3HJSXlDzITTZKTXZVopUruOTm5DN9M\nVcPq0aFDBzWRYepU1aZNVUXc89Spx7ad335T7dlTNSZGdc6cUEZowsGOHapPP63aqpUqqFarpjp4\nsOrChaqHDvkdXdmbOlU1Ls599txHXNzRf1+AVA0ixwZVpy8ivYFngBjgZVV9vNDyJsCrQC2vzGh1\ng6kjImOAm4FDwChVnV/Se1mdvgmUkwPXXQf//S+89lr+wCMm8qhCaipMmeJ+EezZ41r/DB0KN94Y\nuc1BQ9XoIdg6/VKPCrgk/j1wGlAVSANaFiqTBAz3XrcENgW8TgNOABK87cSU9H52pm9yHT6sevPN\n7sxn0iS/ozHlaf9+d6bbs6f7+4uo9uqlOn26ana239GFlkjBs/zch8jRbYcgz/SDqdPvCKxX1Q2q\negCYAfQtVEaBmt7rk4Dc0Uj7AjNU9XdV3Qis97ZnTKnuu891CfDAA66bBRM94uLc/RwLFsCGDfDg\ng+5i8MCB7uLviBHuruEwa3x4TIr7BVNWv2yCSfqNgB8CpjO8eYHGA4NFJAN4Dxh5FOsac4R//MN1\nnHbnnfDww35HY/yUkOB6UN240R0ELr0UXn4ZEhNdL6HPPJN/X0BFVN6DuYSq9c5A4D+q2hi4FHhd\nRILetogME5FUEUnNzMwMUUimonrpJdc18sCBrvdMa8dtwLVs6dnTtWzZvh3+9S83ZOTdd7s7s6+9\n1g2ek5Pjd6RHp7wHcwkmMW8FTg2YbuzNC3QzMBNAVb8AYoF6Qa6LqiapaqKqJta3vlyj2ptvwu23\nwyWXwKuvui+6MYXVrg3Dh7veQb/+2lX3fPwxXHaZS5r33++ahVYUgwa5i7aHD7vnshy9K5ivVArQ\nTEQSRKQqMACYU6jMFqAngIi0wCX9TK/cABE5QUQSgGbAslAFbyLLhx+6G3U6dXLJ3+u3zZgSnXOO\nGzpy61aYNQvatXPVg2ee6cYEeOWV/F5CTRBJX1VzgBHAfGANMFNVV4vIBBHp4xW7F7hVRNKA6cAQ\n74LyatwvgHTgfeBOVT1UFh/EVGxffum6SG7Rwo33WriO05jSVK3qBpB/5x344Qc3VvCOHe5u3wYN\n4OabXV9NkXDx93hY3zvGd6tXQ9eubqi+Tz+1gTpM6Ki6oSKnTIE33nBn/Gee6dr+//nP7lpApAi2\nnb7VmBpfbdrk+tOJjXXVO5bwTSiJwB//6Fr7bN/uqnr+8AcYMwZOPRUuv9zd+HfggN+Rlh9L+sY3\nO3ZAr16ux8wPPnBN84wpKyeeCEOGuK6f161zLcRWrIBrrnFt4p94wvXxFOks6Rtf/PqrG/Vq2zY3\nzOHZZ/sdkYkmzZq5dvBbtrj/v9at3c2ATZvChAn5g8JEIkv6ptxlZcEVV7g+1996y7XWMcYPMTHu\nZq8PPnCNCbp2hYcecsl/zJj8cQAiiSV9U64OHoT+/V0riqlTXX2+MeGgY0d4+203FOSll7pmn/Hx\n7uavrUfcXVRxWdI35ebwYVen+u678MILLvkbE25at3aDvKxZ43p4/b//g9NOczcNbtzod3THz5K+\nKReqrtO0adPgscfcoOnGhLOzznKtfdavd238X3nFXQu48Ub49lu/ozt2lvRNuXj4YXfGdO+9rtWE\nMRVFfLzr52fjRhg1yt0t3rKl+6W6cqXf0R09S/qmzD37rEv6Q4e6njOtAzVTEZ1yiuvuYdMmd5F3\n/nzX5cMVV8DSpX5HFzxL+qZMJSe7ap0rr3Q9B1rCNxVd/fquuefmzfDII+6O306d4MILXadvYdbJ\nwREs6ZsyM3euq//s0QOmT4fKlf2OyJjQqVULxo1zZ/5PPeW6E+nRA7p0gXnzwjf5W9I3ZeLjj13/\n5u3bu2ZwsbF+R2RM2TjxRHetauNGmDwZMjJck8/ERNfFw+HDfkdYkCV9E3KpqdCnD5x+ujvjqVHD\n74iMKXuxsXDHHa4f/ylT3MDu11zjun5OTg6fwV0s6ZuQWrMGeveGunXdXY516/odkTHlq2pV12jh\n229dE+VKlWDwYGje3I357Hfnbpb0Tchs3uw6UKtc2fWY2chGQzZRLCbGDfmZlua6G6ldG265Bc44\nwzVfzs72Jy5L+iYkduxwrRf273dn+Gec4XdExoSHSpVc67Vly+D9912/PiNHul5ln3yy/Hv2tKQf\ngZKT3Q0llSq55+Tksn2/wB4z33vP3cZujClIxH1PliyBTz5x35P/+R/3HS3Pnj0t6UeY5GTXxcHm\nza7J2ObNbrqsEn9WlhuIIj0dZs+2HjONCcYFF+T37NmlS37PnmPHln1TT0v6EWbsWJeIA2Vlufmh\nduCAa53wxRfuglWvXqF/D2MiWeGePb//vuxvYLTbZSLMli1HN/9YHToEN9zg6ihffhn69Qvt9o2J\nJrk9ex46VPbvZWf6EaZJk6ObfyxUYfhwmDnT3Yl4882h27Yx0SwmpuzfI6ikLyK9RWStiKwXkSP6\nSBSR/xWRld5jnYj8GrDsUMCyOaEM3hxp4kSIiys4Ly7OzQ+VMWPgpZfg/vvdnYjGmIqj1OodEYkB\nJgO9gAwgRUTmqGp6bhlVvSeg/EigXcAmslW1behCNiUZNMg9jx3rqnSaNHEJP3f+8frHP9xj+HB4\n9NHQbNMYU36CqdPvCKxX1Q0AIjID6AukF1N+IPBQaMIzx2LQoNAl+UBJSa4v/IED3c0l1mOmMRVP\nMNU7jYAfAqYzvHlHEJGmQAKwMGB2rIikishSEbmymPWGeWVSMzMzgwzdlKc33nDDxV12Gbz6qrsH\nwBhT8YT6qzsAeFNVA69BN1XVROB6YJKInF54JVVNUtVEVU2sX79+iEMyx+v9913fIV26uIu3Var4\nHZEx5lgFk/S3AqcGTDf25hVlADA9cIaqbvWeNwAfU7C+34S5Tz+Fq692Tcrmzj3yIrExpmIJJumn\nAM1EJEFEquIS+xGtcESkOVAb+CJgXm0ROcF7XQ/oTPHXAkyYWbnS3W3bpIk72z/pJL8jMsYcr1Iv\n5KpqjoiMAOYDMcAUVV0tIhOAVFXNPQAMAGaoFriJuAXwoogcxh1gHg9s9WPC17p1rp+QmjXd7eJW\n62ZMZBD5+ZqSAAARC0lEQVQNszG9EhMTNTU11e8wotoPP7j6++xs1znUWWf5HZExpjQisty7floi\n64bBFJCZCRdd5HrO/PhjS/jGRBpL+ibPnj1wySVuoOcPPoB2dsndmIhjSd8AriqnTx/X29/bb0PX\nrn5HZIwpC5b0DQcPQv/+sHix63f/0kv9jsgYU1Ys6Ue5w4fdIM7vvAPPP++6WDDGRC67mT6KqcKo\nUe7s/rHHXDcLxpjIZkk/ij34IEyeDH/7m+tIzRgT+SzpR6mnn3ZdI99yi+sq2XrMNCY6WNKPQlOm\nuMFPrr0WXnjBEr4x0cSSfpT573/h1lvdDVivv14+w7MZY8KHJf0osmCBa51z3nku+Z9wgt8RGWPK\nmyX9KLF0KVx5JTRvDu++C9Wr+x2RMcYPlvSjwKpV7oarBg1g/nyoXdvviIwxfrGkH+G+/97V31er\n5qp3GjTwOyJjjJ/sjtwItm0b9OrlullYvBji4/2OyBjjN0v6EWrnTneGn5kJCxdCy5Z+R2SMCQdW\nvROB0tNdC53162HOHDj3XL8jMsaEC0v6Eeadd+D882HfPneG36OH3xEZY8KJJf0Ioeq6U+jTB5o1\ng5QU+OMf/Y7KGBNurE4/AmRnuz50pk2D665z3SzExfkdlTEmHAV1pi8ivUVkrYisF5Ej+mMUkf8V\nkZXeY52I/Bqw7EYR+c573BjK4A1kZLhRrqZPh4kT3bMlfGNMcUo90xeRGGAy0AvIAFJEZI6qpueW\nUdV7AsqPBNp5r+sADwGJgALLvXV3hfRTRKmlS+Gqq1z9/ezZrmrHGGNKEsyZfkdgvapuUNUDwAyg\nbwnlBwLTvdcXAx+q6i9eov8Q6H08ARvn1VehWzd3Vr90qSV8Y0xwgkn6jYAfAqYzvHlHEJGmQAKw\n8GjWFZFhIpIqIqmZmZnBxB21cnJct8hDhkCXLrBsGbRq5XdUxpiKItStdwYAb6rqoaNZSVWTVDVR\nVRPr168f4pAix6+/wuWXuwFQRo6E99+HunX9jsoYU5EEk/S3AqcGTDf25hVlAPlVO0e7rinB2rXu\nhquFCyEpCZ59FqpU8TsqY0xFE0zSTwGaiUiCiFTFJfY5hQuJSHOgNvBFwOz5wEUiUltEagMXefPM\nUZg3zyX8Xbvgo4/cICjGGHMsSk36qpoDjMAl6zXATFVdLSITRCTw8uEAYIaqasC6vwCP4A4cKcAE\nb54Jgir885+uSic+3t1w1bWr31EZYyoyCcjRYSExMVFTU1P9DsN3v/0Gw4a5IQ2vuca11rGBT4wx\nxRGR5aqaWFo564YhDG3b5ppjvv46PPwwzJxpCd8YExrWDUOYSUlxwxru3u3Gsb3qKr8jMsZEEjvT\nDyPJya7OvkoV+PxzS/jGmNCzpB8GDh2C++6DwYNdt8gpKdC6td9RGWMikVXv+Gz3brj+enjvPbj9\ndmt/b4wpW3amH0LJya5pZaVK7jk5ueTy333nzuw/+AD+9S94/nlL+MaYsmVn+iGSnOyaWGZluenN\nm900wKBBR5b/8EPo3x9iYtzr7t3LLVRjTBSzM/0QGTs2P+Hnyspy8wOpwjPPQO/e0Lixq7+3hG+M\nKS+W9ENky5bS5//+uxvh6u67XVfIn38OCQnlE58xxoAl/ZBp0qTk+T/+CH/6kxvK8IEHYNYsqFGj\n/OIzxhiwpB8yEyceOUxhXJyb/9VXcO65sGKFu7t2wgR3sdcYY8qbpZ4QGTTIdXnctCmIuOekJKhc\n2Q12IgKffQbXXut3pMaYaGZJP4QGDYJNm+DwYdiwAdasgQEDoH17d8G2XTu/IzTGRDtrslkG9u6F\nG26At9+Gm292bfCrVvU7KmOMsaQfctu2wUUXwbffurtrR4xwVTvGGBMOLOmH0P79rinmpk1u/NoL\nL/Q7ImOMKciSfogcPuw6TFuxwlXrWMI3xoQjS/ohMno0zJ4Nkya54Q2NMSYcWeudEHjpJXjySbjj\nDhg1yu9ojDGmeJb0j9OCBTB8uOtL55ln7KKtMSa8WdI/Dunp0K8ftGgBb7zhbsQyxphwFlTSF5He\nIrJWRNaLyOhiyvQXkXQRWS0i0wLmHxKRld5jTqgC99tPP7m6+9hYeOcdqFnT74iMMaZ0pZ6bikgM\nMBnoBWQAKSIyR1XTA8o0A8YAnVV1l4icHLCJbFVtG+K4ffXbb27w8u3b4ZNPXJcLxhhTEQRzpt8R\nWK+qG1T1ADAD6FuozK3AZFXdBaCqP4U2zPChCkOHwhdfwOuvQ8eOfkdkjDHBCybpNwJ+CJjO8OYF\nOhM4U0Q+E5GlItI7YFmsiKR6868s6g1EZJhXJjUzM/OoPkB5Gz8eZsyAv//d1ecbY0xFEqpLj5WB\nZkB3oDGwWETOUdVfgaaqulVETgMWisgqVf0+cGVVTQKSABITEzVEMYXc1KmuW+ShQ+G++/yOxhhj\njl4wZ/pbgVMDpht78wJlAHNU9aCqbgTW4Q4CqOpW73kD8DFQIfuaXLLEdZ7WvTu88II1zTTGVEzB\nJP0UoJmIJIhIVWAAULgVzmzcWT4iUg9X3bNBRGqLyAkB8zsD6VQw69fDVVdBfLwb8cp6zDTGVFSl\nVu+oao6IjADmAzHAFFVdLSITgFRVneMtu0hE0oFDwN9UdaeI/BF4UUQO4w4wjwe2+qkIdu2Cyy5z\nF3DffRfq1PE7ImOMOXaiGl5V6ImJiZqamup3GAAcOODutP30U/joI+ja1e+IjDGmaCKyXFUTSytn\n95AWQ9V1r7BoEbz2miV8Y0xksG4YivGPf8CUKfDAA24ULGOMiQSW9Ivw5pswZowb3/bhh/2Oxhhj\nQseSfiHLlrkz+06d4JVXrGmmMSayWNIPsHmzG+6wYUM3+lVsrN8RGWNMaNmFXM+ePa7XzN9+g4UL\noX59vyMyxpjQs6QP5OTAddfBmjVuQPOWLf2OyBhjykbUJ31VuOsul+yTkmxAc2NMZIv6Ov3nnoN/\n/Qv++le49Va/ozHGmLIV1Un/nXfgnnvcgCiPP+53NMYYU/aiNumnpbl2+G3bui6TY2L8jsgYY8pe\nVCb9bdtcS51atWDuXKhe3e+IjDGmfETdhdz9+11b/F27XEdqp5zid0TGGFN+oirpHz4MgwfDihXu\n5qu2ETVcuzHGlC6qkv5998Hs2TBpkqveMcaYaBM1dfpJSfDUU3DHHTBqlN/RGGOMP6Ii6S9Y4JJ9\n797wzDPWiZoxJnpFfNJPT4d+/aBFC3jjDagcVRVaxhhTUEQn/Z9+cnX3sbHuRqyaNf2OyBhj/BWx\n572//ebutN2+HT75BJo29TsiY4zxX1Bn+iLSW0TWish6ERldTJn+IpIuIqtFZFrA/BtF5DvvcWOo\nAi+JKgwdCl98Aa+/Dh07lse7GmNM+Cv1TF9EYoDJQC8gA0gRkTmqmh5QphkwBuisqrtE5GRvfh3g\nISARUGC5t+6u0H+UfOPHw4wZ8Pe/u/p8Y4wxTjBn+h2B9aq6QVUPADOAvoXK3ApMzk3mqvqTN/9i\n4ENV/cVb9iHQOzShF23qVJgwwZ3p33dfWb6TMcZUPMEk/UbADwHTGd68QGcCZ4rIZyKyVER6H8W6\niMgwEUkVkdTMzMzgoy9kyRK4+Wbo3h1eeMGaZhpjTGGhar1TGWgGdAcGAi+JSK1gV1bVJFVNVNXE\n+sc4TuHGjXDVVRAfD7NmQdWqx7QZY4yJaMEk/a3AqQHTjb15gTKAOap6UFU3AutwB4Fg1g2Jhg2h\nf394912oU6cs3sEYYyq+YJJ+CtBMRBJEpCowAJhTqMxs3Fk+IlIPV92zAZgPXCQitUWkNnCRNy/k\nYmPdCFhnnFEWWzfGmMhQausdVc0RkRG4ZB0DTFHV1SIyAUhV1TnkJ/d04BDwN1XdCSAij+AOHAAT\nVPWXsvggxhhjSieq6ncMBSQmJmpqaqrfYRhjTIUiIstVNbG0chHdDYMxxpiCLOkbY0wUsaRvjDFR\nxJK+McZEEUv6xhgTRSzpG2NMFLGkb4wxUcSSvjHGRBFL+sYYE0Us6RtjTBSxpG+MMVHEkr4xxkQR\nS/rGGBNFLOkbY0wUsaRvjDFRxJK+McZEEUv6xhgTRSzpG2NMFLGkb4wxUcSSvjHGRJGgkr6I9BaR\ntSKyXkRGF7F8iIhkishK73FLwLJDAfPnhDJ4Y4wxR6dyaQVEJAaYDPQCMoAUEZmjqumFir6hqiOK\n2ES2qrY9/lCNMcYcr2DO9DsC61V1g6oeAGYAfcs2LGOMMWUhmKTfCPghYDrDm1fYNSLytYi8KSKn\nBsyPFZFUEVkqIlcW9QYiMswrk5qZmRl89MYYY45KqC7kzgXiVbU18CHwasCypqqaCFwPTBKR0wuv\nrKpJqpqoqon169cPUUjGGGMKCybpbwUCz9wbe/PyqOpOVf3dm3wZ6BCwbKv3vAH4GGh3HPEaY4w5\nDsEk/RSgmYgkiEhVYABQoBWOiDQMmOwDrPHm1xaRE7zX9YDOQOELwMYYY8pJqa13VDVHREYA84EY\nYIqqrhaRCUCqqs4BRolIHyAH+AUY4q3eAnhRRA7jDjCPF9HqxxhjTDkRVfU7hgISExM1NTXV7zCM\nMaZCEZHl3vXTEtkducYYE0Us6RtjTBSxpG+MMVHEkr4xxkSRiEn6yckQHw+VKrnn5GS/IzLGmPBT\napPNiiA5GYYNg6wsN715s5sGGDTIv7iMMSbcRMSZ/tix+Qk/V1aWm2+MMSZfRCT9LVuObr4xxkSr\niEj6TZoc3XxjjIlWEZH0J06EuLiC8+Li3HxjjDH5IiLpDxoESUnQtCmIuOekJLuIa4wxhUVE6x1w\nCd6SvDHGlCwizvSNMcYEx5K+McZEEUv6xhgTRSzpG2NMFLGkb4wxUSTsRs4SkUxg83Fsoh7wc4jC\nqehsXxRk+6Mg2x/5ImFfNFXV+qUVCrukf7xEJDWYIcOige2Lgmx/FGT7I1807Qur3jHGmChiSd8Y\nY6JIJCb9JL8DCCO2Lwqy/VGQ7Y98UbMvIq5O3xhjTPEi8UzfGGNMMSzpG2NMFImYpC8ivUVkrYis\nF5HRfsfjJxE5VUQWiUi6iKwWkbv8jslvIhIjIitE5B2/Y/GbiNQSkTdF5FsRWSMinfyOyU8ico/3\nPflGRKaLSKzfMZWliEj6IhIDTAYuAVoCA0Wkpb9R+SoHuFdVWwLnA3dG+f4AuAtY43cQYeIZ4H1V\nbQ60IYr3i4g0AkYBiap6NhADDPA3qrIVEUkf6AisV9UNqnoAmAH09Tkm36jqdlX9ynu9F/elbuRv\nVP4RkcbAZcDLfsfiNxE5CbgA+DeAqh5Q1V/9jcp3lYFqIlIZiAO2+RxPmYqUpN8I+CFgOoMoTnKB\nRCQeaAd86W8kvpoE/A9w2O9AwkACkAm84lV3vSwi1f0Oyi+quhV4CtgCbAd2q+oH/kZVtiIl6Zsi\niMiJwCzgblXd43c8fhCRy4GfVHW537GEicpAe+B5VW0H7Aei9hqYiNTG1QokAKcA1UVksL9Rla1I\nSfpbgVMDpht786KWiFTBJfxkVf2v3/H4qDPQR0Q24ar9/iQiU/0NyVcZQIaq5v7yexN3EIhWFwIb\nVTVTVQ8C/wX+6HNMZSpSkn4K0ExEEkSkKu5CzByfY/KNiAiuznaNqj7tdzx+UtUxqtpYVeNx/xcL\nVTWiz+RKoqo/Aj+IyFnerJ5Auo8h+W0LcL6IxHnfm55E+IXtiBgYXVVzRGQEMB939X2Kqq72OSw/\ndQZuAFaJyEpv3v2q+p6PMZnwMRJI9k6QNgBDfY7HN6r6pYi8CXyFa/W2ggjvksG6YTDGmCgSKdU7\nxhhjgmBJ3xhjooglfWOMiSKW9I0xJopY0jfGmChiSd8YY6KIJX1jjIki/x9rbcADHh06twAAAABJ\nRU5ErkJggg==\n",
|
|
"text/plain": [
|
|
"<matplotlib.figure.Figure at 0x7f708b9c0048>"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEICAYAAACktLTqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xt8VNW5//HPE+7hfrMqQcBqhXARMEU9SCGIFm/wQ6nl\nZsVqUVul9dKWg5dalFP1UEQ9HFv0V0+rEcoLj3cs7a9iUWuRQBUKSKHcDKACCoioGHh+f6xJMgm5\nDGQyO5l836/XvDJ7z9p7P7OTPLNm7bXXMndHRETSS0bUAYiISPIpuYuIpCEldxGRNKTkLiKShpTc\nRUTSkJK7iEgaUnKXcplZAzPbb2YnJbNslMzsFDNLet9fMxtmZpvjlteZ2aBEyh7DsR4zs6nHun0l\n+73HzP4n2fuV6DSMOgBJDjPbH7eYCXwBHIotX+vueUezP3c/BLRIdtn6wN1PS8Z+zOwaYIK7D4nb\n9zXJ2LekPyX3NOHuxck1VjO8xt3/X0XlzayhuxemIjYRST01y9QTsa/dvzezuWb2CTDBzM42s7+Z\n2R4z22FmD5lZo1j5hmbmZtY1tvxk7PWXzewTM3vTzLodbdnY6xeY2T/NbK+ZPWxmb5jZxAriTiTG\na81sg5l9bGYPxW3bwMweMLPdZrYRGF7J+bnNzOaVWTfbzGbGnl9jZmtj7+dfsVp1RfsqMLMhseeZ\nZvZELLbVwBllyt5uZhtj+11tZiNi63sD/wUMijV57Yo7t3fFbX9d7L3vNrNnzeyERM5NVcxsVCye\nPWb2ipmdFvfaVDPbbmb7zOzduPd6lpmtiK3/wMz+M9HjSQ1wdz3S7AFsBoaVWXcPcBC4hPCh3gz4\nOnAm4RvcycA/gRti5RsCDnSNLT8J7AJygEbA74Enj6HsccAnwMjYazcDXwITK3gvicT4HNAa6Ap8\nVPTegRuA1UAW0B5YEv7kyz3OycB+oHncvj8EcmLLl8TKGDAU+AzoE3ttGLA5bl8FwJDY8xnAq0Bb\noAuwpkzZy4ETYr+TcbEYvhJ77Rrg1TJxPgncFXt+fizGvkBT4L+BVxI5N+W8/3uA/4k97xGLY2js\ndzQVWBd73hPYAhwfK9sNODn2fBkwNva8JXBm1P8L9fmhmnv98rq7v+Duh939M3df5u5L3b3Q3TcC\nc4DBlWy/wN3z3f1LII+QVI627MXA2+7+XOy1BwgfBOVKMMZfuPted99MSKRFx7oceMDdC9x9N3Bv\nJcfZCPyD8KEDcB7wsbvnx15/wd03evAK8Geg3IumZVwO3OPuH7v7FkJtPP648919R+x38hThgzkn\ngf0CjAcec/e33f1zYAow2Myy4spUdG4qMwZ43t1fif2O7iV8QJwJFBI+SHrGmvY2xc4dhA/pU82s\nvbt/4u5LE3wfUgOU3OuX9+IXzKy7mb1kZu+b2T5gGtChku3fj3t+gMovolZU9sT4ONzdCTXdciUY\nY0LHItQ4K/MUMDb2fFxsuSiOi81sqZl9ZGZ7CLXmys5VkRMqi8HMJprZO7Hmjz1A9wT3C+H9Fe/P\n3fcBHwOd4socze+sov0eJvyOOrn7OuAWwu/hw1gz3/GxolcB2cA6M3vLzC5M8H1IDVByr1/KdgP8\nNaG2eoq7twLuJDQ71KQdhGYSAMzMKJ2MyqpOjDuAznHLVXXVnA8MM7NOhBr8U7EYmwELgF8Qmkza\nAH9MMI73K4rBzE4GHgGuB9rH9vtu3H6r6ra5ndDUU7S/loTmn20JxHU0+80g/M62Abj7k+4+kNAk\n04BwXnD3de4+htD09kvgaTNrWs1Y5BgpuddvLYG9wKdm1gO4NgXHfBHob2aXmFlD4IdAxxqKcT7w\nIzPrZGbtgZ9WVtjd3wdeB/4HWOfu62MvNQEaAzuBQ2Z2MXDuUcQw1czaWLgP4Ia411oQEvhOwufc\n9wg19yIfAFlFF5DLMRe42sz6mFkTQpJ9zd0r/CZ0FDGPMLMhsWP/mHCdZKmZ9TCz3NjxPos9DhPe\nwBVm1iFW098be2+HqxmLHCMl9/rtFuBKwj/urwkXPmuUu38AfBuYCewGvgr8ndAvP9kxPkJoG19F\nuNi3IIFtniJcIC1uknH3PcBNwDOEi5KjCR9SifgZ4RvEZuBl4Hdx+10JPAy8FStzGhDfTv0nYD3w\ngZnFN68Ubf8HQvPIM7HtTyK0w1eLu68mnPNHCB88w4ERsfb3JsD9hOsk7xO+KdwW2/RCYK2F3lgz\ngG+7+8HqxiPHxkKTp0g0zKwBoRlgtLu/FnU8IulCNXdJOTMbHmumaALcQehl8VbEYYmkFSV3icI5\nwEbCV/5vAqPcvaJmGRE5BmqWERFJQ6q5i4ikocgGDuvQoYN37do1qsOLiNRJy5cv3+XulXUfBiJM\n7l27diU/Pz+qw4uI1ElmVtWd1oCaZURE0pKSu4hIGlJyFxFJQ5qJSaSe+PLLLykoKODzzz+POhRJ\nQNOmTcnKyqJRo4qGFqqckrtIPVFQUEDLli3p2rUrYTBOqa3cnd27d1NQUEC3bt2q3qAcdapZJi8P\nunaFjIzwM++opnwWqd8+//xz2rdvr8ReB5gZ7du3r9a3rDpTc8/Lg0mT4MCBsLxlS1gGGF/tcfBE\n6gcl9rqjur+rOlNzv+22ksRe5MCBsF5EREqrM8l969ajWy8itcvu3bvp27cvffv25fjjj6dTp07F\nywcPJjbs+1VXXcW6desqLTN79mzyktRme8455/D2228nZV+pVmeaZU46KTTFlLdeRJIvLy98M966\nNfyfTZ9evSbQ9u3bFyfKu+66ixYtWnDrrbeWKuPuuDsZGeXXOx9//PEqj/ODH/zg2INMI3Wm5j59\nOmRmll6XmRnWi0hyFV3j2rIF3EuucdVEJ4YNGzaQnZ3N+PHj6dmzJzt27GDSpEnk5OTQs2dPpk2b\nVly2qCZdWFhImzZtmDJlCqeffjpnn302H374IQC33347s2bNKi4/ZcoUBgwYwGmnncZf//pXAD79\n9FMuu+wysrOzGT16NDk5OVXW0J988kl69+5Nr169mDp1KgCFhYVcccUVxesfeughAB544AGys7Pp\n06cPEyZMSPo5S0SdqbkX1RiSWZMQkfJVdo2rJv7n3n33XX73u9+Rk5MDwL333ku7du0oLCwkNzeX\n0aNHk52dXWqbvXv3MnjwYO69915uvvlmfvOb3zBlypQj9u3uvPXWWzz//PNMmzaNP/zhDzz88MMc\nf/zxPP3007zzzjv079+/0vgKCgq4/fbbyc/Pp3Xr1gwbNowXX3yRjh07smvXLlatWgXAnj17ALj/\n/vvZsmULjRs3Ll6XanWm5g7hj2rzZjh8OPxUYhepGam+xvXVr361OLEDzJ07l/79+9O/f3/Wrl3L\nmjVrjtimWbNmXHDBBQCcccYZbN68udx9X3rppUeUef311xkzZgwAp59+Oj179qw0vqVLlzJ06FA6\ndOhAo0aNGDduHEuWLOGUU05h3bp1TJ48mUWLFtG6dWsAevbsyYQJE8jLyzvmm5Cqq04ldxFJjYqu\nZdXUNa7mzZsXP1+/fj0PPvggr7zyCitXrmT48OHl9vdu3Lhx8fMGDRpQWFhY7r6bNGlSZZlj1b59\ne1auXMmgQYOYPXs21157LQCLFi3iuuuuY9myZQwYMIBDhw4l9biJUHIXkSNEeY1r3759tGzZklat\nWrFjxw4WLVqU9GMMHDiQ+fPnA7Bq1apyvxnEO/PMM1m8eDG7d++msLCQefPmMXjwYHbu3Im7861v\nfYtp06axYsUKDh06REFBAUOHDuX+++9n165dHCjbxpUCdabNXURSJ8prXP379yc7O5vu3bvTpUsX\nBg4cmPRj3HjjjXznO98hOzu7+FHUpFKerKws7r77boYMGYK7c8kll3DRRRexYsUKrr76atwdM+O+\n++6jsLCQcePG8cknn3D48GFuvfVWWrZsmfT3UJXI5lDNyclxTdYhkjpr166lR48eUYdRKxQWFlJY\nWEjTpk1Zv349559/PuvXr6dhw9pV3y3vd2Zmy909p4JNitWudyIikgL79+/n3HPPpbCwEHfn17/+\nda1L7NWVXu9GRCQBbdq0Yfny5VGHUaN0QVVEJA0puYuIpCEldxGRNKTkLiKShpTcRSQlcnNzj7gh\nadasWVx//fWVbteiRQsAtm/fzujRo8stM2TIEKrqWj1r1qxSNxNdeOGFSRn35a677mLGjBnV3k+y\nKbmLSEqMHTuWefPmlVo3b948xo4dm9D2J554IgsWLDjm45dN7gsXLqRNmzbHvL/aTsldRFJi9OjR\nvPTSS8UTc2zevJnt27czaNCg4n7n/fv3p3fv3jz33HNHbL9582Z69eoFwGeffcaYMWPo0aMHo0aN\n4rPPPisud/311xcPF/yzn/0MgIceeojt27eTm5tLbm4uAF27dmXXrl0AzJw5k169etGrV6/i4YI3\nb95Mjx49+N73vkfPnj05//zzSx2nPG+//TZnnXUWffr0YdSoUXz88cfFxy8aArhowLK//OUvxZOV\n9OvXj08++eSYz2151M9dpB760Y8g2RMM9e0LsbxYrnbt2jFgwABefvllRo4cybx587j88ssxM5o2\nbcozzzxDq1at2LVrF2eddRYjRoyocB7RRx55hMzMTNauXcvKlStLDdk7ffp02rVrx6FDhzj33HNZ\nuXIlkydPZubMmSxevJgOHTqU2tfy5ct5/PHHWbp0Ke7OmWeeyeDBg2nbti3r169n7ty5PProo1x+\n+eU8/fTTlY7P/p3vfIeHH36YwYMHc+edd/Lzn/+cWbNmce+997Jp0yaaNGlS3BQ0Y8YMZs+ezcCB\nA9m/fz9NmzY9irNdNdXcRSRl4ptm4ptk3J2pU6fSp08fhg0bxrZt2/jggw8q3M+SJUuKk2yfPn3o\n06dP8Wvz58+nf//+9OvXj9WrV1c5KNjrr7/OqFGjaN68OS1atODSSy/ltddeA6Bbt2707dsXqHxY\nYQjjy+/Zs4fBgwcDcOWVV7JkyZLiGMePH8+TTz5ZfCfswIEDufnmm3nooYfYs2dP0u+QVc1dpB6q\nrIZdk0aOHMlNN93EihUrOHDgAGeccQYAeXl57Ny5k+XLl9OoUSO6du1a7jC/Vdm0aRMzZsxg2bJl\ntG3blokTJx7TfooUDRcMYcjgqpplKvLSSy+xZMkSXnjhBaZPn86qVauYMmUKF110EQsXLmTgwIEs\nWrSI7t27H3OsZanmLiIp06JFC3Jzc/nud79b6kLq3r17Oe6442jUqBGLFy9mS3kTJsf5xje+wVNP\nPQXAP/7xD1auXAmE4YKbN29O69at+eCDD3j55ZeLt2nZsmW57dqDBg3i2Wef5cCBA3z66ac888wz\nDBo06KjfW+vWrWnbtm1xrf+JJ55g8ODBHD58mPfee4/c3Fzuu+8+9u7dy/79+/nXv/5F7969+elP\nf8rXv/513n333aM+ZmVUcxeRlBo7diyjRo0q1XNm/PjxXHLJJfTu3ZucnJwqa7DXX389V111FT16\n9KBHjx7F3wBOP/10+vXrR/fu3encuXOp4YInTZrE8OHDOfHEE1m8eHHx+v79+zNx4kQGDBgAwDXX\nXEO/fv0qbYKpyG9/+1uuu+46Dhw4wMknn8zjjz/OoUOHmDBhAnv37sXdmTx5Mm3atOGOO+5g8eLF\nZGRk0LNnz+JZpZIloSF/zWw48CDQAHjM3e8t8/oDQG5sMRM4zt0r7WOkIX9FUktD/tY9NTrkr5k1\nAGYD5wEFwDIze97di69SuPtNceVvBPolHr6IiCRbIm3uA4AN7r7R3Q8C84CRlZQfC8xNRnAiInJs\nEknunYD34pYLYuuOYGZdgG7AK9UPTUSSLaqZ1+ToVfd3lezeMmOABe5e7lTfZjbJzPLNLH/nzp1J\nPrSIVKZp06bs3r1bCb4OcHd2795drRubEuktsw3oHLecFVtXnjHADyrakbvPAeZAuKCaYIwikgRZ\nWVkUFBSgilXd0LRpU7Kyso55+0SS+zLgVDPrRkjqY4BxZQuZWXegLfDmMUcjIjWmUaNGdOvWLeow\nJEWqbJZx90LgBmARsBaY7+6rzWyamY2IKzoGmOf6ziciErmEbmJy94XAwjLr7iyzfFfywhIRkeqo\nc8MP5OfDL38Jf/87HD4cdTQiIrVTnRt+4E9/gqlTw/N27WDIEMjNhaFDoUcPqGCEUBGReiWh4Qdq\nQnWGH9i2DRYvhldeCY+iMYa+8pWSRD90KJx8spK9iKSXRIcfqJPJvaxNm0oS/eLFsGNHWN+5c0mi\nz80NyyIidVm9Su7x3GHdupJk/+qrsHt3eO2UU0oSfW5uqOmLiNQliSb3OndBtSpm0L07fP/7sGAB\nfPhhmE5s5szQJj9vHowdC8cfD716weTJ8Mwz8NFHiR8jLw+6doWMjPAzL6+m3o2IyLFJu5p7VQoL\nYcWKkjb7116Dzz4LHwr9+pW02Q8aBC1bHrl9Xh5MmgRxk6iTmQlz5sD48al7HyJSP9XbZpmjdfAg\nvPVWSTPOm2+GdQ0awNe/XtJm/2//Bs2ahZp6eZPEdOkCxzC2v4jIUVFyP0YHDoQEX5Tsly2DQ4eg\ncWM4+2z4y1/K385M/e5FpOYlbbKO+iYzE849NzwA9u2D118vSfYVOemk1MQnIpKItLugmmytWsGF\nF8KMGaGt/le/CrX4eI0bwz33RBOfiEh5lNyP0rXXwm9+E9rYAZo0CW30jz8OGzdGG5uISBEl92Mw\nfny4eOoe2uh/9avQNt+rFzzwQGijFxGJkpJ7NWVkhNr8mjWhV83NN8PAgbB6ddSRiUh9puSeJFlZ\n8MILoR/8hg2hz/y0aaHJRkQk1ZTck8gMxo2DtWvhssvgZz+DnJzQZCMikkpK7jWgY0eYOxeeey6M\na3PWWfDjH5e+q1VEpCYpudegESNCW/w114SulH36hIHMRERqmpJ7DWvdGn7965IboHJzwwXYvXuj\njUtE0puSe4rk5sLKlXDLLfDYY9CzJ7z4YtRRiUi6UnJPoczM0Dzz5pvQti1cckm4ALtzZ9SRiUi6\nUXKPwIABsHw53HVXGHM+OztcgI1oDDcRSUNK7hFp3Dh0lVyxIsz1Om5cuABbUBB1ZCKSDpTcI9ar\nF/z1r2GmqD//ObTFz5mT2PDBmhFKRCqi5F4LNGgAN90Eq1bBGWeE3jTnnhvudK1I0YxQW7aE5pwt\nW8KyEryIgJJ7rfLVr4ba+6OPhuaaPn3gl78MUwOWddttR94UdeBAWC8iouRey5iFm57WrIHzzoNb\nbw1T/K1aVbrc1q3lb1/RehGpX5Tca6lOneDZZ2HevDC8cP/+4QLsF1+E1yua+UkzQokIJJjczWy4\nma0zsw1mNqWCMpeb2RozW21mTyU3zPrJDL797VCLHzMmjDJ5xhmwdClMnx76zcfLzAzrRUSqTO5m\n1gCYDVwAZANjzSy7TJlTgX8HBrp7T+BHNRBrvdWhAzzxRLijde/eMFH38uXw0ENhRiiz8HPOnDCR\niIhIIhNkDwA2uPtGADObB4wE1sSV+R4w290/BnD3D5MdqMBFF4VJQKZMCTM+nXxymPJv6NCoIxOR\n2iaRZplOwHtxywWxdfG+BnzNzN4ws7+Z2fDydmRmk8ws38zyd+qe+2PSqhX893+H0SUzMkKXyalT\no45KRGqbZF1QbQicCgwBxgKPmlmbsoXcfY6757h7TseOHZN06Ppp8OAwENnVV8MvfgEPPhh1RCJS\nmyTSLLMN6By3nBVbF68AWOruXwKbzOyfhGSvOYhqULNmYTjhjz4KN0GddBKMGhV1VCJSGyRSc18G\nnGpm3cysMTAGeL5MmWcJtXbMrAOhmWZjEuOUCjRoAE8+GQYjGzcu9KQREakyubt7IXADsAhYC8x3\n99VmNs3MRsSKLQJ2m9kaYDHwY3ffXVNBS2mZmfD883DiiWEY4X/9K+qIRCRq5hGNM5uTk+P5+fmR\nHDtd/fOfoZtk+/ZhzPj27aOOSESSzcyWu3tOVeV0h2oa+drXwqTcW7fCyJHw+edRRyQiUVFyTzPn\nnAO//S288QZceWViQweLSPpJpLeM1DHf/naovf/kJ2Gc9/vuizoiEUk1Jfc0deutsGkT3H9/SPDX\nXx91RCKSSkruacosjD2zdSvccAN07gwXXxx1VCKSKmpzT2MNG4Yhg/v2DU01y5dHHZGIpIqSe5pr\n0SKMJtmhQ6i5b9kSdUQikgpK7vXACSfAyy/DZ5/BBRfAxx9HHZGI1DQl93oiOxueeSZMun3ppSUz\nOolIelJyr0dyc8P476++GuZpjejmZBFJAfWWqWcmTAhzst5xR+gieffdUUckIjVByb0euu22kODv\nuQe6dYPvfjfqiEQk2ZTc6yEzeOQRKCiASZOgUyf45jejjkpEkklt7vVUo0Ywfz707AmjR8M770Qd\nkYgkk5J7PdaqFbz0ErRuHSbfLiiIOiIRSRYl93ouKwsWLoR9+0KC37cv6ohEJBmU3IU+feDpp2HN\nmtBE8+WXUUckItWl5C4AnHcezJkDf/oTXHut+sCL1HXqLSPFrroqDBN8992hi+Qdd0QdkYgcKyV3\nKeXnPw994O+8M9zkdMUVUUckIsdCyV1KMYPHHoNt2+Dqq0Mf+KFDo45KRI6W2tzlCI0bhwusX/ta\nGGRs9eqoIxKRo6XkLuVq0yb0gW/WDC68EHbsiDoiETkaSu5SoS5dQoLfvTtM9LF/f/nl8vJC+3xG\nRviZl5fKKEWkPEruUqn+/cMwBe+8E6bqKyws/XpeXhifZsuW0H1yy5awrAQvEi0ld6nShRfC7Nnh\nTtYbbyzdB/622+DAgdLlDxwI60UkOuotIwm59trQB/6++0LTy09/GtZv3Vp++YrWi0hqqOYuCfuP\n/4AxY2DKFJg3L6w76aTyy1a0XkRSI6HkbmbDzWydmW0wsynlvD7RzHaa2duxxzXJD1WilpEBjz8O\ngwbBlVfCa6/B9OmQmVm6XGZmWC8i0akyuZtZA2A2cAGQDYw1s+xyiv7e3fvGHo8lOU6pJZo2hWef\nDcMTjBwJOTlhTJouXcINUF26hOXx46OOVKR+S6TNfQCwwd03ApjZPGAksKYmA5Paq127cHH1rLPg\nggvgzTeVzEVqm0SaZToB78UtF8TWlXWZma00swVm1rm8HZnZJDPLN7P8nTt3HkO4UlucfDK8+CK8\n/z6MGHFkjxkRiVayLqi+AHR19z7An4DfllfI3ee4e46753Ts2DFJh5aoDBgAc+fCsmUwbhwcOhR1\nRCJSJJHkvg2Ir4lnxdYVc/fd7v5FbPEx4IzkhCe13ciR8OCD8NxzMHEiHDwYdUQiAom1uS8DTjWz\nboSkPgYYF1/AzE5w96LRR0YAa5MapdRqN94Ypue7/fbQTPP002F+VhGJTpU1d3cvBG4AFhGS9nx3\nX21m08xsRKzYZDNbbWbvAJOBiTUVsNROt90WukkuXgyDB8P27VFHJFK/mUc0n1pOTo7n5+dHcmyp\nOYsWwWWXQfv28Ic/QI8eUUckkl7MbLm751RVTneoSlJ985uwZAl88QUMHAivvx51RCL1k5K7JF3/\n/qHve8eOMGxYaIMXkdRScpca0a0bvPFGSPTf+hY8/HDUEYnUL0ruUmM6dIA//zl0l5w8GX7yEzh8\nOOqoROoHJXepUc2awYIF8P3vw3/+J0yYENrjRaRmaTx3qXENGsB//VcYBnjKlNAX/plnoHXrqCMT\nSV+quUtKmIUJPp54IgwVPGgQFBREHZVI+lJyl5SaMAFefhk2b4azz4Z//CPqiETSk5K7pNywYaEv\n/KFDcM458Je/RB2RSPpRcpdI9O0b+sKfeCKcfz78/vdRRySSXpTcJTJduoQ7WM88M8zNOnNm1BGJ\npA8ld4lUu3bwxz/C6NFwyy1w003qCy+SDEruErmmTUOzzA9/CLNmhVr8559HHZVI3aZ+7lIrZGTA\nAw9A585w663wwQdhIu62baOOTKRuUs1dag2z0DQzdy787W+hJ83WrVFHJVI3KblLrTNmTBgXftu2\n0Bf+nXeijkik7lFyl1ppyJBwJ6tZuJv1z3+OOiKRukXJXWqt3r1D80yXLnDBBZCXF3VEInWHkrvU\nallZoQY/cGAYuuC++yCimSFF6hT1lpFar02bMB/rxIlhVMn33oMHHwyjTYpUZe9eWLs2PPbvr7q8\nWfXLVPX64MHQs2fVx6kOJXepE5o0Cc0yWVkwYwZs3x6WmzWLOjKpDdxh586QwNesKf1z+/aoozvS\nI48ouYsUy8gIE35kZYU7WYcNg+efh/bto45MUsU9DBVdXhLfvbukXIsWkJ0N550XfvboER5V3TeR\nSJNfVWUS2UeLFlWXqS4ld6lzfvhD6NQptMEPHBiGEO7WLeqoJJkOHYJNm8pP4vFNK+3bh+R92WUl\nSTw7O/x9JNK8ks6U3KVOGj0avvIVGDEi9IVfuDBMxi11y8GDsH59SeIuSuLr1pWejvHEE0PSvuqq\n0km8Y8foYq/tlNylzho0CN54I3STHDw4zNX6zW9GHZWU59NPQ8IuWwvfsCHU0iHUtLt2DUn7/PNL\nN6doSsajp+QudVp2dhgX/sIL4eKL4dFH4cor9ZU8Vb74AnbsCPPi7thx5CN+fZGGDeGUU8IFxdGj\nS5L4aadBZmZ07yXdmEfUaTgnJ8fz8/MjObakn3374NJLw52sjRqFJpv4x/HHl7/ctq0+CMpyh08+\nqTpZ79gBH3985PYZGXDccXDCCSWPLl1Kkvgpp0Djxql/X+nCzJa7e05V5VRzl7TQqhVccQX8/e/w\n0Uch6bRrFxLQ22/Dhx9CYeGR2zVqFBJR2eRf3gdCu3Z1+4Pg8GHYtaviRB2//sCBI7dv0iQk6uOP\nD7XsIUNKJ/Ci1447Tvcg1AYJJXczGw48CDQAHnP3eysodxmwAPi6u6taLimTlwff/35JUvr009Ce\nO2cOjB8fEttHH4WhhIse779fejnRD4Kqvg0UfRAcOgRffpnY4+DB5JYrKvvppyVJ+4MPyn9frVqV\nJOczzyxJ0mUTd5s2dfvDrb6pslnGzBoA/wTOAwqAZcBYd19TplxL4CWgMXBDVcldzTKSTF27wpYt\nR67v0gU2UiEYAAAHaElEQVQ2bz66fR0+HGr+ZZN/2eWiR3kJM5UaNw4fPOU9mjcvP1HHJ3C1c9ct\nyWyWGQBscPeNsR3PA0YCa8qUuxu4D/jxUcYqUm0Vjft+LOPBZ2SE/tPt21d9F2HRB0HZD4CPPgoX\nDitKuo0aVZ6UEy3ToIFq01K+RJJ7J+C9uOUC4Mz4AmbWH+js7i+ZWYXJ3cwmAZMATjrppKOPVqQC\nJ51Ufs29pv/M4j8IsrNr9lgiR6Pao0KaWQYwE7ilqrLuPsfdc9w9p6PuPpAkmj79yOaFzMywXqQ+\nSiS5bwM6xy1nxdYVaQn0Al41s83AWcDzZlZlm5BIsowfHy6edukSmim6dCm5mCpSHyXSLLMMONXM\nuhGS+hhgXNGL7r4X6FC0bGavAreqt4yk2vjxSuYiRaqsubt7IXADsAhYC8x399VmNs3MRtR0gCIi\ncvQS6ufu7guBhWXW3VlB2SHVD0tERKpD0+yJiKQhJXcRkTSk5C4ikoaU3EVE0pCSu4hIGlJyFxFJ\nQ0ruIiJpSMldRCQNKbmLiKQhJXcRkTSk5C4ikoaU3EWSKC8vTPmXkRF+5uVFHZHUVwkNHCYiVcvL\ng0mTSibp3rIlLIOGIpbUU81dJEluu60ksRc5cCCsF0k1JXeRJEnmJN0i1aXkLpIkFU3GrbngJQpK\n7iJJokm6pTZRchdJEk3SLbWJesuIJJEm6ZbaQjV3EZE0pOQuIpKGlNxFRNKQkruISBpSchcRSUNK\n7iIiaUjJXSQNaXRKUT93kTSj0SkFVHMXSTsanVIgweRuZsPNbJ2ZbTCzKeW8fp2ZrTKzt83sdTPL\nTn6oIpIIjU4pkEByN7MGwGzgAiAbGFtO8n7K3Xu7e1/gfmBm0iMVkYRodEqBxGruA4AN7r7R3Q8C\n84CR8QXcfV/cYnPAkxeiiBwNjU4pkFhy7wS8F7dcEFtXipn9wMz+Rai5Ty5vR2Y2yczyzSx/586d\nxxKviFRBo1MKJPGCqrvPdvevAj8Fbq+gzBx3z3H3nI4dOybr0CJSxvjxsHkzHD4cfiqx1z+JJPdt\nQOe45azYuorMA/5PdYISEZHqSSS5LwNONbNuZtYYGAM8H1/AzE6NW7wIWJ+8EEVE5GhVmdzdvRC4\nAVgErAXmu/tqM5tmZiNixW4ws9Vm9jZwM3BljUUsInWG7pSNjrlH07ElJyfH8/PzIzm2iNS8snfK\nQui1o4u71WNmy909p6pyukNVRGqE7pSNlpK7iNQI3SkbLSV3EakRulM2WkruIlIjdKdstJTcRaRG\n6E7ZaGk8dxGpMePHK5lHRTV3EZE0pOQuIpKGlNxFJO3Vxztl1eYuImmtvs4pq5q7iKS1+nqnrJK7\niKS1+nqnrJK7iKS1+nqnrJK7iKS1+nqnrJK7iKS1+nqnrJK7iKS92jKnbCq7ZKorpIhICqS6S6Zq\n7iIiKZDqLplK7iIiKZDqLplK7iIiKZDqLplK7iIiKZDqLplK7iIiKZDqLpnqLSMikiKpnLxENXcR\nkTSk5C4ikoaU3EVE0pCSu4hIGlJyFxFJQ+bu0RzYbCew5Rg37wDsSmI4dZ3OR2k6HyV0LkpLh/PR\nxd07VlUosuReHWaW7+45UcdRW+h8lKbzUULnorT6dD7ULCMikoaU3EVE0lBdTe5zog6gltH5KE3n\no4TORWn15nzUyTZ3ERGpXF2tuYuISCWU3EVE0lCdS+5mNtzM1pnZBjObEnU8UTGzzma22MzWmNlq\nM/th1DHVBmbWwMz+bmYvRh1L1MysjZktMLN3zWytmZ0ddUxRMbObYv8n/zCzuWbWNOqYalqdSu5m\n1gCYDVwAZANjzSw72qgiUwjc4u7ZwFnAD+rxuYj3Q2Bt1EHUEg8Cf3D37sDp1NPzYmadgMlAjrv3\nAhoAY6KNqubVqeQODAA2uPtGdz8IzANGRhxTJNx9h7uviD3/hPCP2ynaqKJlZlnARcBjUccSNTNr\nDXwD+L8A7n7Q3fdEG1WkGgLNzKwhkAlsjzieGlfXknsn4L245QLqeUIDMLOuQD9gabSRRG4W8BPg\ncNSB1ALdgJ3A47FmqsfMrHnUQUXB3bcBM4CtwA5gr7v/Mdqoal5dS+5Shpm1AJ4GfuTu+6KOJypm\ndjHwobsvjzqWWqIh0B94xN37AZ8C9fIalZm1JXzD7wacCDQ3swnRRlXz6lpy3wZ0jlvOiq2rl8ys\nESGx57n7/0YdT8QGAiPMbDOhuW6omT0ZbUiRKgAK3L3o29wCQrKvj4YBm9x9p7t/Cfwv8G8Rx1Tj\n6lpyXwacambdzKwx4aLI8xHHFAkzM0J76lp3nxl1PFFz93939yx370r4u3jF3dO+dlYRd38feM/M\nToutOhdYE2FIUdoKnGVmmbH/m3OpBxeX69QE2e5eaGY3AIsIV7x/4+6rIw4rKgOBK4BVZvZ2bN1U\nd18YYUxSu9wI5MUqQhuBqyKOJxLuvtTMFgArCL3M/k49GIZAww+IiKShutYsIyIiCVByFxFJQ0ru\nIiJpSMldRCQNKbmLiKQhJXcRkTSk5C4ikob+PyYufzJSbaZ7AAAAAElFTkSuQmCC\n",
|
|
"text/plain": [
|
|
"<matplotlib.figure.Figure at 0x7f707af3d5c0>"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
}
|
|
],
|
|
"source": [
|
|
"import matplotlib.pyplot as plt\n",
|
|
"\n",
|
|
"acc = history.history['acc']\n",
|
|
"val_acc = history.history['val_acc']\n",
|
|
"loss = history.history['loss']\n",
|
|
"val_loss = history.history['val_loss']\n",
|
|
"\n",
|
|
"epochs = range(len(acc))\n",
|
|
"\n",
|
|
"plt.plot(epochs, acc, 'bo', label='Training acc')\n",
|
|
"plt.plot(epochs, val_acc, 'b', label='Validation acc')\n",
|
|
"plt.title('Training and validation accuracy')\n",
|
|
"plt.legend()\n",
|
|
"\n",
|
|
"plt.figure()\n",
|
|
"\n",
|
|
"plt.plot(epochs, loss, 'bo', label='Training loss')\n",
|
|
"plt.plot(epochs, val_loss, 'b', label='Validation loss')\n",
|
|
"plt.title('Training and validation loss')\n",
|
|
"plt.legend()\n",
|
|
"\n",
|
|
"plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Combining CNNs and RNNs to process long sequences\n",
|
|
"\n",
|
|
"\n",
|
|
"Because 1D convnets process input patches independently, they are not sensitive to the order of the timesteps (beyond a local scale, the \n",
|
|
"size of the convolution windows), unlike RNNs. Of course, in order to be able to recognize longer-term patterns, one could stack many \n",
|
|
"convolution layers and pooling layers, resulting in upper layers that would \"see\" long chunks of the original inputs -- but that's still a \n",
|
|
"fairly weak way to induce order-sensitivity. One way to evidence this weakness is to try 1D convnets on the temperature forecasting problem \n",
|
|
"from the previous section, where order-sensitivity was key to produce good predictions. Let's see:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"metadata": {
|
|
"collapsed": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# We reuse the following variables defined in the last section:\n",
|
|
"# float_data, train_gen, val_gen, val_steps\n",
|
|
"\n",
|
|
"import os\n",
|
|
"import numpy as np\n",
|
|
"\n",
|
|
"data_dir = '/home/ubuntu/data/'\n",
|
|
"fname = os.path.join(data_dir, 'jena_climate_2009_2016.csv')\n",
|
|
"\n",
|
|
"f = open(fname)\n",
|
|
"data = f.read()\n",
|
|
"f.close()\n",
|
|
"\n",
|
|
"lines = data.split('\\n')\n",
|
|
"header = lines[0].split(',')\n",
|
|
"lines = lines[1:]\n",
|
|
"\n",
|
|
"float_data = np.zeros((len(lines), len(header) - 1))\n",
|
|
"for i, line in enumerate(lines):\n",
|
|
" values = [float(x) for x in line.split(',')[1:]]\n",
|
|
" float_data[i, :] = values\n",
|
|
" \n",
|
|
"mean = float_data[:200000].mean(axis=0)\n",
|
|
"float_data -= mean\n",
|
|
"std = float_data[:200000].std(axis=0)\n",
|
|
"float_data /= std\n",
|
|
"\n",
|
|
"def generator(data, lookback, delay, min_index, max_index,\n",
|
|
" shuffle=False, batch_size=128, step=6):\n",
|
|
" if max_index is None:\n",
|
|
" max_index = len(data) - delay - 1\n",
|
|
" i = min_index + lookback\n",
|
|
" while 1:\n",
|
|
" if shuffle:\n",
|
|
" rows = np.random.randint(\n",
|
|
" min_index + lookback, max_index, size=batch_size)\n",
|
|
" else:\n",
|
|
" if i + batch_size >= max_index:\n",
|
|
" i = min_index + lookback\n",
|
|
" rows = np.arange(i, min(i + batch_size, max_index))\n",
|
|
" i += len(rows)\n",
|
|
"\n",
|
|
" samples = np.zeros((len(rows),\n",
|
|
" lookback // step,\n",
|
|
" data.shape[-1]))\n",
|
|
" targets = np.zeros((len(rows),))\n",
|
|
" for j, row in enumerate(rows):\n",
|
|
" indices = range(rows[j] - lookback, rows[j], step)\n",
|
|
" samples[j] = data[indices]\n",
|
|
" targets[j] = data[rows[j] + delay][1]\n",
|
|
" yield samples, targets\n",
|
|
" \n",
|
|
"lookback = 1440\n",
|
|
"step = 6\n",
|
|
"delay = 144\n",
|
|
"batch_size = 128\n",
|
|
"\n",
|
|
"train_gen = generator(float_data,\n",
|
|
" lookback=lookback,\n",
|
|
" delay=delay,\n",
|
|
" min_index=0,\n",
|
|
" max_index=200000,\n",
|
|
" shuffle=True,\n",
|
|
" step=step, \n",
|
|
" batch_size=batch_size)\n",
|
|
"val_gen = generator(float_data,\n",
|
|
" lookback=lookback,\n",
|
|
" delay=delay,\n",
|
|
" min_index=200001,\n",
|
|
" max_index=300000,\n",
|
|
" step=step,\n",
|
|
" batch_size=batch_size)\n",
|
|
"test_gen = generator(float_data,\n",
|
|
" lookback=lookback,\n",
|
|
" delay=delay,\n",
|
|
" min_index=300001,\n",
|
|
" max_index=None,\n",
|
|
" step=step,\n",
|
|
" batch_size=batch_size)\n",
|
|
"\n",
|
|
"# This is how many steps to draw from `val_gen`\n",
|
|
"# in order to see the whole validation set:\n",
|
|
"val_steps = (300000 - 200001 - lookback) // batch_size\n",
|
|
"\n",
|
|
"# This is how many steps to draw from `test_gen`\n",
|
|
"# in order to see the whole test set:\n",
|
|
"test_steps = (len(float_data) - 300001 - lookback) // batch_size"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 1/20\n",
|
|
"500/500 [==============================] - 124s - loss: 0.4189 - val_loss: 0.4521\n",
|
|
"Epoch 2/20\n",
|
|
"500/500 [==============================] - 11s - loss: 0.3629 - val_loss: 0.4545\n",
|
|
"Epoch 3/20\n",
|
|
"500/500 [==============================] - 11s - loss: 0.3399 - val_loss: 0.4527\n",
|
|
"Epoch 4/20\n",
|
|
"500/500 [==============================] - 11s - loss: 0.3229 - val_loss: 0.4721\n",
|
|
"Epoch 5/20\n",
|
|
"500/500 [==============================] - 11s - loss: 0.3122 - val_loss: 0.4712\n",
|
|
"Epoch 6/20\n",
|
|
"500/500 [==============================] - 11s - loss: 0.3030 - val_loss: 0.4705\n",
|
|
"Epoch 7/20\n",
|
|
"500/500 [==============================] - 11s - loss: 0.2935 - val_loss: 0.4870\n",
|
|
"Epoch 8/20\n",
|
|
"500/500 [==============================] - 11s - loss: 0.2862 - val_loss: 0.4676\n",
|
|
"Epoch 9/20\n",
|
|
"500/500 [==============================] - 11s - loss: 0.2817 - val_loss: 0.4738\n",
|
|
"Epoch 10/20\n",
|
|
"500/500 [==============================] - 11s - loss: 0.2775 - val_loss: 0.4896\n",
|
|
"Epoch 11/20\n",
|
|
"500/500 [==============================] - 11s - loss: 0.2715 - val_loss: 0.4765\n",
|
|
"Epoch 12/20\n",
|
|
"500/500 [==============================] - 11s - loss: 0.2683 - val_loss: 0.4724\n",
|
|
"Epoch 13/20\n",
|
|
"500/500 [==============================] - 11s - loss: 0.2644 - val_loss: 0.4842\n",
|
|
"Epoch 14/20\n",
|
|
"500/500 [==============================] - 11s - loss: 0.2606 - val_loss: 0.4910\n",
|
|
"Epoch 15/20\n",
|
|
"500/500 [==============================] - 11s - loss: 0.2558 - val_loss: 0.5000\n",
|
|
"Epoch 16/20\n",
|
|
"500/500 [==============================] - 11s - loss: 0.2539 - val_loss: 0.4960\n",
|
|
"Epoch 17/20\n",
|
|
"500/500 [==============================] - 11s - loss: 0.2516 - val_loss: 0.4875\n",
|
|
"Epoch 18/20\n",
|
|
"500/500 [==============================] - 11s - loss: 0.2501 - val_loss: 0.4884\n",
|
|
"Epoch 19/20\n",
|
|
"500/500 [==============================] - 11s - loss: 0.2444 - val_loss: 0.5024\n",
|
|
"Epoch 20/20\n",
|
|
"500/500 [==============================] - 11s - loss: 0.2444 - val_loss: 0.4821\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"from keras.models import Sequential\n",
|
|
"from keras import layers\n",
|
|
"from keras.optimizers import RMSprop\n",
|
|
"\n",
|
|
"model = Sequential()\n",
|
|
"model.add(layers.Conv1D(32, 5, activation='relu',\n",
|
|
" input_shape=(None, float_data.shape[-1])))\n",
|
|
"model.add(layers.MaxPooling1D(3))\n",
|
|
"model.add(layers.Conv1D(32, 5, activation='relu'))\n",
|
|
"model.add(layers.MaxPooling1D(3))\n",
|
|
"model.add(layers.Conv1D(32, 5, activation='relu'))\n",
|
|
"model.add(layers.GlobalMaxPooling1D())\n",
|
|
"model.add(layers.Dense(1))\n",
|
|
"\n",
|
|
"model.compile(optimizer=RMSprop(), loss='mae')\n",
|
|
"history = model.fit_generator(train_gen,\n",
|
|
" steps_per_epoch=500,\n",
|
|
" epochs=20,\n",
|
|
" validation_data=val_gen,\n",
|
|
" validation_steps=val_steps)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Here are our training and validation Mean Absolute Errors:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 5,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEICAYAAACzliQjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmcFNW5//HPwyaCCAgoyjagXGFwYZmQ5CIiahJwATXE\ni2IkLhc1GjWaXPkJSRTjK+JCjF6iEoMbo6MXoyEmSlyIaBKRARFkU0CUAWQTQXaGeX5/nBpohll6\npnu6Z6a/79erX9NVdar66eqep06fOnXK3B0REckM9dIdgIiIpI6SvohIBlHSFxHJIEr6IiIZRElf\nRCSDKOmLiGQQJX2pFDOrb2bbzKxjMsumk5mdYGZJ77tsZmeb2cqY6aVm1j+eslV4rcfN7Paqrl/O\ndn9tZk8me7uSPg3SHYBULzPbFjPZBNgN7Iumr3H33Mpsz933AUcku2wmcPcTk7EdM7sauMzdz4jZ\n9tXJ2LbUfUr6dZy770+6UU3yand/o6zyZtbA3QtTEZuIpJ6adzJc9PP9eTN7zsy+Bi4zs2+b2Xtm\n9pWZrTWzh8ysYVS+gZm5mWVF01Oi5a+a2ddm9m8z61zZstHywWb2sZltMbOHzeyfZvajMuKOJ8Zr\nzGyZmW02s4di1q1vZr81s01mtgIYVM7+GWNmeSXmTTSzCdHzq81scfR+lke18LK2VWBmZ0TPm5jZ\nM1FsC4E+JcqONbMV0XYXmtmQaP7JwP8C/aOms40x+/aOmPWvjd77JjN72cyOjWffVMTMLozi+crM\n3jKzE2OW3W5ma8xsq5ktiXmv3zKzudH8dWZ2X7yvJ9XA3fXIkAewEji7xLxfA3uA8wmVgMOBbwDf\nJPwS7AJ8DNwQlW8AOJAVTU8BNgI5QEPgeWBKFcoeDXwNDI2W3QLsBX5UxnuJJ8Y/A82BLODL4vcO\n3AAsBNoDrYCZ4V+h1NfpAmwDmsZsez2QE02fH5Ux4ExgJ3BKtOxsYGXMtgqAM6Ln9wP/AFoCnYBF\nJcpeDBwbfSaXRjEcEy27GvhHiTinAHdEz78bxdgTaAz8Hngrnn1Tyvv/NfBk9Lx7FMeZ0Wd0O7A0\net4D+AxoG5XtDHSJns8GLomeNwO+me7/hUx+qKYvAO+6+1/cvcjdd7r7bHef5e6F7r4CmAQMKGf9\nqe6e7+57gVxCsqls2fOAee7+52jZbwkHiFLFGeNv3H2Lu68kJNji17oY+K27F7j7JuCecl5nBfAR\n4WAE8B1gs7vnR8v/4u4rPHgLeBMo9WRtCRcDv3b3ze7+GaH2Hvu6L7j72ugzeZZwwM6JY7sAI4DH\n3X2eu+8CRgMDzKx9TJmy9k15hgPT3P2t6DO6h3Dg+CZQSDjA9IiaCD+N9h2Eg3dXM2vl7l+7+6w4\n34dUAyV9AVgVO2Fm3czsr2b2hZltBcYBrctZ/4uY5zso/+RtWWWPi43D3Z1QMy5VnDHG9VqEGmp5\nngUuiZ5fGk0Xx3Gemc0ysy/N7CtCLbu8fVXs2PJiMLMfmdmHUTPKV0C3OLcL4f3t3567bwU2A+1i\nylTmMytru0WEz6iduy8FbiV8Duuj5sK2UdErgGxgqZm9b2bnxPk+pBoo6QuEn/uxHiPUbk9w9yOB\nXxKaL6rTWkJzCwBmZhycpEpKJMa1QIeY6Yq6lL4AnG1m7Qg1/mejGA8HpgK/ITS9tAD+HmccX5QV\ng5l1AR4BrgNaRdtdErPdirqXriE0GRVvrxmhGWl1HHFVZrv1CJ/ZagB3n+Lu/QhNO/UJ+wV3X+ru\nwwlNeA8AL5pZ4wRjkSpS0pfSNAO2ANvNrDtwTQpe8xWgt5mdb2YNgJuANtUU4wvAzWbWzsxaAbeV\nV9jdvwDeBZ4Elrr7J9Giw4BGwAZgn5mdB5xViRhuN7MWFq5juCFm2RGExL6BcPz7b0JNv9g6oH3x\nietSPAdcZWanmNlhhOT7jruX+cupEjEPMbMzotf+OeE8zCwz625mA6PX2xk9ighv4Idm1jr6ZbAl\nem9FCcYiVaSkL6W5FRhJ+Id+jHDCtVq5+zrgv4AJwCbgeOADwnUFyY7xEULb+wLCScapcazzLOHE\n7P6mHXf/Cvgp8BLhZOgwwsErHr8i/OJYCbwKPB2z3fnAw8D7UZkTgdh28NeBT4B1ZhbbTFO8/muE\nZpaXovU7Etr5E+LuCwn7/BHCAWkQMCRq3z8MuJdwHuYLwi+LMdGq5wCLLfQOux/4L3ffk2g8UjUW\nmk5FahYzq09oThjm7u+kOx6RukI1fakxzGxQ1NxxGPALQq+P99MclkidoqQvNclpwApC08H3gAvd\nvazmHRGpAjXviIhkENX0RUQySI0bcK1169aelZWV7jBERGqVOXPmbHT38ro5AzUw6WdlZZGfn5/u\nMEREahUzq+jKckDNOyIiGUVJX0Qkgyjpi4hkECV9EZEMoqQvIpJBlPRFRDKIkr6ISAaJK+lHA2Et\njW6kPLqU5T8ysw1mNi96XB2zbKSZfRI9RiYzeBGRmuLVV+GNN9IdRcUqTPrRELcTgcGEW55dYmbZ\npRR93t17Ro/Ho3WPIowb/k2gL/ArM2uZtOhFRNJs0ya49FI45xz43vfgmWfSHVH54rkity+wrPgm\nx2aWR7hl3KI41v0e8Lq7fxmt+zrhxgvPVS1cEamJ1qyBP/4Rtm2DXr2gZ0/o2hXq1093ZNVr2jS4\n5hrYuBF+9St4910YORL27oUrr0x3dKWLJ+m34+AbOBcQau4lfd/MTgc+Bn7q7qvKWPeQ+56a2Shg\nFEDHjhXdrlREagJ3+Pe/4eGHYepU2LcPGjQICQ+gSRM45ZRwECg+EJx8MjSuA3fH3bwZbr4Znn46\nvMdXXw3vb+dOuPBCuOoq2L0brrsu3ZEeKllj7/wFeM7dd5vZNcBTwJnxruzuk4BJADk5ORrrWaQG\n27ULnn8+JPs5c6B5c7jxRrj+emjfHhYvhg8+gHnzwt/cXHjkkbBu/frQvXtIkLEHg5a1qNH31Vfh\n6qth3Tr4xS9g7Fho1CgsO/xwePll+MEP4Mc/hj174Kab0htvSfEk/dVAh5jp9tG8/dx9U8zk44R7\nZRave0aJdf9R2SBFJP1Wrw7Je9Ik2LABsrPD9GWXwRFHHCh36qnhUayoCFauDAeA4oPBW2/BlCkH\nynTqFJL/d74Ttte8ecreVty2bIFbbw3NWD16hKadPn0OLde4Mbz4IlxySfg1sGcP/PznqY+3TO5e\n7oNwYFgBdAYaAR8CPUqUOTbm+YXAe9Hzo4BPCTdJbhk9P6q81+vTp4+LSM1QVOT+7rvuF1/sXr++\nu5n7kCHub7wRliVi3Tr36dPd77nHffhw965d3cG9SRP3q65ynz07Oe8hGf7+d/cOHdzr1XMfPdp9\n166K19mzJ7wvcL/rruqPEcj3CvK5h3DiKBTuZv8xsBwYE80bBwyJnv8GWBgdEGYA3WLWvRJYFj2u\nqOi1lPRF0m/nTvcnnnDv1StkiRYt3G+91X358up93dmz3a++OiR+cO/Tx/0Pf3Dftq16X7csW7e6\nX3NNiKVbN/f33qvc+nv3uv/wh2H9X/wi8QNleZKa9FP5UNIXSZ9Vq9xvv929deuQHXr0cH/ssdQn\n3a++cp840f2kk0IcRx7pfv317vPnpy6GN99079Qp/Lr52c/cd+yo2nYKC92vvDK8j9tuq77Er6Qv\nInHZssU9N9d96NDQhFOvnvsFF7i/9Vb11kzjUVTk/s9/htryYYeFjNWvn/szz4RfI9Vh27ZwgIHQ\n5PTPfya+zX373K+7Lmzz5purZ78q6YtUwtdfh3/Ghx8ONbO6bvNm96eecj//fPdGjUImOO4495//\n3P3TT9MdXek2bnR/4IEDbf9HHRWanJYuTd5rzJzp3qVLqN3ffLP79u3J23ZRkftNN4XYf/zjcCBI\npniTvoWyNUdOTo7rdomSSgsXwrBhsGRJmM7JCT1UevVKb1zJ9uWXoTvh1KlhuIC9e6FDh/Dehw2D\nb30L6tWC0bjcYcYMePRReOklKCyEs86Ca6+Fb3879JXfvh127Kjc3y1bQq+izp3hiSfg9NOrJ/bb\nboP77gvdPh97LHn73MzmuHtOHEGkv3Yf+1BNv3bautX9pz91f+WVdEdSOU895X744e7HHBOaM557\nzv3oo0Mzx623pu8EYrKsX+8+aZL7d7/r3qBBqGVmZYUa/axZ6W++SdTate533x3a3kNKje9h5n7E\nEeGz7tw5nLvo29f9llvCr77qVFTkPnZsiGPkyOT9skQ1fUmVJUvCVYhLloRay+OPwxVXpDuq8u3c\nCT/5SehzPWAAPPccHHtsWLZ5c6iN/eEPof/4738fxlWpLb74ItSAp06Ff/wj9JM/4YQDNfrevcEs\n3VEm17598Prr8Nln0LRpuBq4vL+HHZb+fXDXXfDLX4b+/E8/Ha5mToRq+pISf/qTe7Nm7m3ahFr+\nd74TajAPPJDuyMr28cfup5wS4hwzJnSrK83Mme7du4dyF18capU11d697n/8o/vpp4daLLifeGKo\nUc6bV/tr9HXVPfeEz2rYsNCvPxHoRK5Up8LC0LUPws/iVavC/F27whe4OKHWtGTzwgvhINWqlfvf\n/lZx+V273MeNCz1Hmjd3f/TR5J+AS9Rrr4XmCXDPzna/4w73jz6qefteSjdhQvjshgyJ76Kvsijp\nS7XZuDG0EYP7qFGHflELC8MFNhC6qdWEJLl7t/tPfhJi+ta33D//vHLrL13qPnBgWP8//zMk1XRb\nuNB98OAQU5cu7i++qERfW/3v/4bPcfDgqrfxK+lLtZg7N5wIbNQoXClZlqKicCEKhEvRd+9OXYwl\nrVwZfo1AONlc1ViKityffDL8SmjYMPySqeoFO4nYsCF0+atfP1y0dN99idUQpWaYNCl8llWlpC9J\n99RT7o0bu7dvH3p+xKO4zXLw4OT2eY7XX/7i3rJlSI5/+lNytrlhg/vll4f3dcIJYRyaVNi1y/3+\n+0MzU/36IfGvX5+a15aaT0lfkmb37gNXKA4cGAbKqoxJk8LJxX79wkVBqbB374FfGr16uS9blvzX\neOONkPQhHASqKwEXFYWmm+OPP3AAXbiwel5Lai8lfUmK1atDGzaE8UfK6ulSkRdeCE0ip57q/sUX\nyY2xpIIC9/79Q8zXXFN9l+u7h+adMWPCe2vVyv1//ie81+XLk9O+PmeO+4ABvn8cnNdeS3ybUjcp\n6UvC3nnHvW1b96ZN3Z9/PvHtTZ8eRk884YTqu9T/9ddD99GmTd2nTKme1yjNRx+5n312SP7FFwC1\nbOl+1lnhQJCX5/7JJ/EfCFavDhfumIX388gjVT/gSmaIN+nr4iw5hDtMnAg//Wm4JP1Pf4KTTkrO\ntv/973ChU5Mm4WKa7OzEt+kOH34YbsoxYUK4M9PUqeFvqu3eDQsWhDtKzZ0b/i5YEG6kAeHmIL17\nh5tvFP894YQDl+Lv2AH33w/jx4fhBW6+GW6/vWbeVERqlngvzlLSr0GWLg03WG7a9OBHkybQsGFq\nYtixI4xh8swzcP754UrBFi2S+xoLFsB3vxsS4auvQt++ld/G9u3w5pvw17+Gx+roXm4jR4YDVtOm\nyY05EXv2wEcfHXwgmD8/HCAAjjwyjPNz0klhbJzVq8OVs+PHQ5cu6Y1dag8l/VrmscfCTZTL+jga\nNjz0YBB7UCh+fvjh4dG48YG/ZT0vOW/zZhgxItSa77wTxoypvgG4VqwIt8Zbtw7+/OcwYFZFVq4M\nCf6VV8KAW7t3Q7Nm4QBy3nkweDAcc0z1xJtse/eGgd7mzDlwMPjww3CT7QcegNNOS3eEUtso6dcS\n7vCb34QEe+654QbT27eX/SgeEbCsx65dYVyZwsKqxdOiRbiRdSrGmlm7NiTsjz+GvLwwfk+swkL4\n178OJPpFi8L8rl1Dkj/3XOjf/8BNqWu7oqLaMcql1EzxJv0Eh/iRRBQVhRsmT5gQathPPJG8ZpzC\nwlAT3rkzHAiKDwaxf0s+37MnNOlkZSUnhooceyy8/XZI3sOGhQHOhg6F114Lif6118KvjwYNwqBo\nV18dyv7Hf6QmvlRTwpdUUE0/TQoL4b//G558Moz2+OCDmftPv307XHQR/P3vYR8UFcHRR4dfG+ed\nF5qBjjwy3VGK1Gyq6ddgu3bB8OGhLfuOO8Lwquke5jWdmjaFadNCM1dRUUj0OTmZexAUqU5K+im2\ndStccEE4EfnQQ6GWL2F88zvuSHcUInWfkn4KbdgQepjMmxf6lI8Yke6IRCTTKOmnyOefh54qn30W\nmnXOPTfdEYlIJlLST4ElS8LJyK1bw8nK/v3THZGIZCol/WqWnx+adOrVC90Te/ZMd0QiksnUP6Ia\nzZgBAweG3invvquELyLpp6RfTV5+OdTwO3aEf/4zXEUqIpJuSvrV4Mkn4fvfDzX7mTOhXbt0RyQi\nEmRcm757uLR/1arwKCg48HzNmjB4WZs2Bx5HH33wdJs2YXCyskyYALfeCmefDS+9BEcckbr3JiJS\nkbiSvpkNAn4H1Aced/d7yij3fWAq8A13zzezLGAxsDQq8p67X5to0OXZuvVAEi+Z1IsfO3YcvE79\n+nDccaFGvnFjGPVww4YwEmJpmjYt/WCwaRNMnhzGkZkyJVxwJCJSk1SY9M2sPjAR+A5QAMw2s2nu\nvqhEuWbATcCsEptY7u7VfgpzzZpw04ytWw+ebxYG9mrfPoxXPngwdOhw8KNt25D4Y7mHba1fHw4A\nsY/YeWvWhCFx168PA5Zdc00Yz73k9kREaoJ4avp9gWXuvgLAzPKAocCiEuXuAsYDP09qhHFq0wYu\nv/zQhH7ccVUbudIs3K2oefP4TsK6hzF1ymv6ERFJt3iSfjtgVcx0AfDN2AJm1hvo4O5/NbOSSb+z\nmX0AbAXGuvs7JV/AzEYBowA6duxYifAPaNgQHn64SqsmhZkSvojUfAn33jGzesAE4NZSFq8FOrp7\nL+AW4FkzO2SQXHef5O457p7Tpk2bREMSEZEyxJP0VwMdYqbbR/OKNQNOAv5hZiuBbwHTzCzH3Xe7\n+yYAd58DLAfq6C0wRERqvniS/mygq5l1NrNGwHBgWvFCd9/i7q3dPcvds4D3gCFR75020YlgzKwL\n0BVYkfR3ISIicamwTd/dC83sBmA6ocvmZHdfaGbjgHx3n1bO6qcD48xsL1AEXOvuXyYjcBERqTzd\nLlFEpA6I93aJGoZBRCSDKOmLiGQQJX0RkQyipC8ikkGU9EVEMoiSvohIBlHSFxHJIEr6IiIZRElf\nRCSDKOmLiGQQJX0RkQyipC8ikkGU9EVEMoiSvohIBlHSFxHJIHUm6efmQlYW1KsX/ubmpjsiEZGa\np8I7Z9UGubkwahTs2BGmP/ssTAOMGJG+uEREapo6UdMfM+ZAwi+2Y0eYLyIiB9SJpP/555WbLyKS\nqepE0u/YsXLzRUQyVZ1I+nffDU2aHDyvSZMwX0REDqgTSX/ECJg0CTp1ArPwd9IkncQVESmpTvTe\ngZDgleRFRMpXJ2r6IiISHyV9EZEMoqQvIpJBlPRFRDKIkr6ISAZR0hcRySBK+iIiGSSupG9mg8xs\nqZktM7PR5ZT7vpm5meXEzPt/0XpLzex7yQhaRESqpsKLs8ysPjAR+A5QAMw2s2nuvqhEuWbATcCs\nmHnZwHCgB3Ac8IaZ/Ye770veWxARkXjFU9PvCyxz9xXuvgfIA4aWUu4uYDywK2beUCDP3Xe7+6fA\nsmh7IiKSBvEk/XbAqpjpgmjefmbWG+jg7n+t7LrR+qPMLN/M8jds2BBX4CIiUnkJn8g1s3rABODW\nqm7D3Se5e46757Rp0ybRkEREpAzxDLi2GugQM90+mlesGXAS8A8zA2gLTDOzIXGsKyIiKRRPTX82\n0NXMOptZI8KJ2WnFC919i7u3dvcsd88C3gOGuHt+VG64mR1mZp2BrsD7SX8XIiISlwpr+u5eaGY3\nANOB+sBkd19oZuOAfHefVs66C83sBWARUAhcr547IiLpY+6e7hgOkpOT4/n5+ekOQ0SkVjGzOe6e\nU1E5XZErIpJBlPRFRDKIkr6ISAZR0hcRySBK+iIiGURJX0Qkgyjpi4hkECV9EZEMoqQvIpJBlPRF\nRDKIkr6ISAZR0hcRySBK+iIiGURJX0Qkgyjpi4hkECV9EZEMoqQvIpJBlPRFRDKIkr6ISAZR0o/k\n5kJWFtSrF/7m5qY7IhGR5GuQ7gBqgtxcGDUKduwI0599FqYBRoxIX1wiIsmmmj4wZsyBhF9sx44w\nX0SkLlHSBz7/vHLzRURqKyV9oGPHys0XEamtlPSBu++GJk0OntekSZgvIlKXKOkTTtZOmgSdOoFZ\n+Dtpkk7iikjdo947kREjlORFpO5TTV9EJIMo6YuIZJC4kr6ZDTKzpWa2zMxGl7L8WjNbYGbzzOxd\nM8uO5meZ2c5o/jwzezTZb0BEROJXYZu+mdUHJgLfAQqA2WY2zd0XxRR71t0fjcoPASYAg6Jly929\nZ3LDFhGRqoinpt8XWObuK9x9D5AHDI0t4O5bYyabAp68EEVEJFniSfrtgFUx0wXRvIOY2fVmthy4\nF7gxZlFnM/vAzN42s/6lvYCZjTKzfDPL37BhQyXCFxGRykjaiVx3n+juxwO3AWOj2WuBju7eC7gF\neNbMjixl3UnunuPuOW3atElWSCIiUkI8SX810CFmun00ryx5wAUA7r7b3TdFz+cAy4H/qFqoIiKS\nqHiS/mygq5l1NrNGwHBgWmwBM+saM3ku8Ek0v010Ihgz6wJ0BVYkI3AREam8CnvvuHuhmd0ATAfq\nA5PdfaGZjQPy3X0acIOZnQ3sBTYDI6PVTwfGmdleoAi41t2/rI43IiIiFTP3mtXRJicnx/Pz89Md\nRqXl5obx9z//PIzOeffdGtZBRFLHzOa4e05F5TT2ThLozlsiUltoGIYk0J23RKS2UNJPAt15S0Rq\nCyX9JNCdt0SktlDSTwLdeUtEagsl/STQnbdEpLZQ750k0Z23RKQ2UE1fRCSDKOmLiGQQJX0RkQyi\npC8ikkGU9EVEMoiSfg2RmwtZWVCvXvibm5vuiESkLlKXzRpAA7aJSKqopl8DaMA2EUkVJf0aQAO2\niUiqKOnXABqwTURSRUm/BtCAbSKSKkr6NYAGbBORVFHvnRpCA7aJSCqopl9HqJ+/iMRDNf06QP38\nRSRequnXAernLyLxUtKvA9TPX0TipaRfB6ifv4jES0m/DlA/fxGJl5J+HaB+/iISLyX9OmLECFi5\nEoqKwt/KJnx1+RTJDOqyKeryKZJB4qrpm9kgM1tqZsvMbHQpy681swVmNs/M3jWz7Jhl/y9ab6mZ\nfS+ZwUtyqMunSOaoMOmbWX1gIjAYyAYuiU3qkWfd/WR37wncC0yI1s0GhgM9gEHA76PtSQ2iLp8i\nmSOemn5fYJm7r3D3PUAeMDS2gLtvjZlsCnj0fCiQ5+673f1TYFm0PalB1OVTJHPEk/TbAatipgui\neQcxs+vNbDmhpn9jJdcdZWb5Zpa/YcOGeGOXJFGXT5HMkbTeO+4+0d2PB24DxlZy3UnunuPuOW3a\ntElWSBIndfkUyRzxJP3VQIeY6fbRvLLkARdUcV1JE3X5FMkM8ST92UBXM+tsZo0IJ2anxRYws64x\nk+cCn0TPpwHDzewwM+sMdAXeTzxsqUmKu3x+9hm4H+jyqcQvUvNUmPTdvRC4AZgOLAZecPeFZjbO\nzIZExW4ws4VmNg+4BRgZrbsQeAFYBLwGXO/u+6rhfUgaqcunSO1h7l5xqRTKycnx/Pz8dIchlVCv\nXqjhl2QWmotEpPqZ2Rx3z6monIZhkISpy6dI7aGkLwlTl0+R2kNJXxKWjC6f6v0jkhoacE2SYsSI\nqvfr14BvIqmjmr6knXr/iKSOkr6kXTIGfFPzkEh8lPQl7RLt/aOLw0Tip6QvaZdo7x81D4nET0lf\n0i7R3j+6H4BI/GpF7529e/dSUFDArl270h2KxKFx48a0b9+ehg0bxr1OIr1/OnYMTTqlzY9Xbm74\nZfD552G9u+9WzyGpm2pF0i8oKKBZs2ZkZWVhZukOR8rh7mzatImCggI6d+6ckte8++6Du3xC5ZqH\n1GVUMkmtaN7ZtWsXrVq1UsKvBcyMVq1apfRXWaLNQzonIJmkVtT0ASX8WiQdn1UizUM6JyCZpFbU\n9EWqUzIGjNN1AlJb1Mmkn+x/wE2bNtGzZ0969uxJ27Ztadeu3f7pPXv2xLWNK664gqVLl5ZbZuLE\nieQmKVucdtppzJs3LynbqusS7TKq6wSkVnH3GvXo06ePl7Ro0aJD5pVlyhT3Jk3cw79feDRpEuYn\nw69+9Su/7777DplfVFTk+/btS86LJEG/fv38gw8+SNvrV+YzqwmmTHHv1MndLPytzPelU6eDv2/F\nj06dqidWkdIA+R5Hjq1zNf1UnpRbtmwZ2dnZjBgxgh49erB27VpGjRpFTk4OPXr0YNy4cfvLFte8\nCwsLadGiBaNHj+bUU0/l29/+NuvXrwdg7NixPPjgg/vLjx49mr59+3LiiSfyr3/9C4Dt27fz/e9/\nn+zsbIYNG0ZOTk6FNfopU6Zw8sknc9JJJ3H77bcDUFhYyA9/+MP98x966CEAfvvb35Kdnc0pp5zC\nZZddlvR9VlMlco9gDSMhtUmtOZEbr1SflFuyZAlPP/00OTnhhjX33HMPRx11FIWFhQwcOJBhw4aR\nnZ190DpbtmxhwIAB3HPPPdxyyy1MnjyZ0aNHH7Jtd+f9999n2rRpjBs3jtdee42HH36Ytm3b8uKL\nL/Lhhx/Su3fvcuMrKChg7Nix5Ofn07x5c84++2xeeeUV2rRpw8aNG1mwYAEAX331FQD33nsvn332\nGY0aNdo/T8qX6HUC6jIqqVTnavqpvovT8ccfvz/hAzz33HP07t2b3r17s3jxYhYtWnTIOocffjiD\nBw8GoE+fPqxcubLUbV900UWHlHn33XcZPnw4AKeeeio9evQoN75Zs2Zx5pln0rp1axo2bMill17K\nzJkzOeGEE1i6dCk33ngj06dPp3nz5gD06NGDyy67jNzc3EpdXJXJNIyE1CZ1Lumn+i5OTZs23f/8\nk08+4XektVlrAAAOCElEQVS/+x1vvfUW8+fPZ9CgQaX2V2/UqNH+5/Xr16ewsLDUbR922GEVlqmq\nVq1aMX/+fPr378/EiRO55pprAJg+fTrXXnsts2fPpm/fvuzbp/vYV6QmDCOh5iGJV51L+sm4i1NV\nbd26lWbNmnHkkUeydu1apk+fnvTX6NevHy+88AIACxYsKPWXRKxvfvObzJgxg02bNlFYWEheXh4D\nBgxgw4YNuDs/+MEPGDduHHPnzmXfvn0UFBRw5plncu+997Jx40Z2lKyCSqkSOSegUUYllepcmz4k\ndqFOInr37k12djbdunWjU6dO9OvXL+mv8ZOf/ITLL7+c7Ozs/Y/ippnStG/fnrvuuoszzjgDd+f8\n88/n3HPPZe7cuVx11VW4O2bG+PHjKSws5NJLL+Xrr7+mqKiIn/3sZzRr1izp70EOlugwEuU1D+mc\ngBwini4+qXwk2mWzrtu7d6/v3LnT3d0//vhjz8rK8r1796Y5qkPpM6ucRLqMmpXeZdQsNa8vNQNx\ndtmskzX9umzbtm2cddZZFBYW4u489thjNGigj7G2S+coo+o9lFmULWqZFi1aMGfOnHSHITWImoek\nMurciVyRTKPeQ1IZqumL1AFqHpJ4qaYvkuFqwsVl+qWQOkr6Ihku3c1DybjOQAeN+MWV9M1skJkt\nNbNlZnbIIDFmdouZLTKz+Wb2ppl1ilm2z8zmRY9pyQw+VQYOHHjIhVYPPvgg1113XbnrHXHEEQCs\nWbOGYcOGlVrmjDPOID8/v9ztPPjggwddJHXOOeckZVycO+64g/vvvz/h7Ujtl86LyxL9paCL0yqn\nwqRvZvWBicBgIBu4xMyySxT7AMhx91OAqcC9Mct2unvP6DEkSXGn1CWXXEJeXt5B8/Ly8rjkkkvi\nWv+4445j6tSpVX79kkn/b3/7Gy1atKjy9kSSKdHmoUR/Kah5qXLiOZHbF1jm7isAzCwPGArsv/7f\n3WfElH8PqLYxeW++GZJ9b5CePSEa0bhUw4YNY+zYsezZs4dGjRqxcuVK1qxZQ//+/dm2bRtDhw5l\n8+bN7N27l1//+tcMHTr0oPVXrlzJeeedx0cffcTOnTu54oor+PDDD+nWrRs7d+7cX+66665j9uzZ\n7Ny5k2HDhnHnnXfy0EMPsWbNGgYOHEjr1q2ZMWMGWVlZ5Ofn07p1ayZMmMDkyZMBuPrqq7n55ptZ\nuXIlgwcP5rTTTuNf//oX7dq1489//jOHH354me9x3rx5XHvttezYsYPjjz+eyZMn07JlSx566CEe\nffRRGjRoQHZ2Nnl5ebz99tvcdNNNQLg14syZM3XlbgYr/lUwZkxI1B07hoQf76+FRE8kJ6t5KVNO\nRMfTvNMOWBUzXRDNK8tVwKsx043NLN/M3jOzC6oQY9odddRR9O3bl1dfDW8rLy+Piy++GDOjcePG\nvPTSS8ydO5cZM2Zw6623Ei6OK90jjzxCkyZNWLx4MXfeeedBfe7vvvtu8vPzmT9/Pm+//Tbz58/n\nxhtv5LjjjmPGjBnMmDHjoG3NmTOHJ554glmzZvHee+/xhz/8gQ8++AAIg79df/31LFy4kBYtWvDi\niy+W+x4vv/xyxo8fz/z58zn55JO58847gTBU9AcffMD8+fN59NFHAbj//vuZOHEi8+bN45133in3\nYCKZIZHmoUR/KaS7eam2SWqXTTO7DMgBBsTM7uTuq82sC/CWmS1w9+Ul1hsFjALoWMEnVV6NvDoV\nN/EMHTqUvLw8/vjHPwJhGIvbb7+dmTNnUq9ePVavXs26deto27ZtqduZOXMmN954IwCnnHIKp5xy\nyv5lL7zwApMmTaKwsJC1a9eyaNGig5aX9O6773LhhRfuH+nzoosu4p133mHIkCF07tyZnj17AuUP\n3wxhfP+vvvqKAQPCxzZy5Eh+8IMf7I9xxIgRXHDBBVxwQThm9+vXj1tuuYURI0Zw0UUX0b59+3h2\noUipEv2lkOjFaam+B0e6xVPTXw10iJluH807iJmdDYwBhrj77uL57r46+rsC+AfQq+S67j7J3XPc\nPadNmzaVegOpMnToUN58803mzp3Ljh076NOnDwC5ubls2LCBOXPmMG/ePI455phSh1OuyKeffsr9\n99/Pm2++yfz58zn33HOrtJ1ixcMyQ2JDM//1r3/l+uuvZ+7cuXzjG9+gsLCQ0aNH8/jjj7Nz5076\n9evHkiVLqhynCCT2SyHR3kfJuAdHoucEUnlOIZ6kPxvoamadzawRMBw4qBeOmfUCHiMk/PUx81ua\n2WHR89ZAP2LOBdQmRxxxBAMHDuTKK6886ATuli1bOProo2nYsCEzZszgs9IaJ2OcfvrpPPvsswB8\n9NFHzJ8/HwjDMjdt2pTmzZuzbt26/U1JAM2aNePrr78+ZFv9+/fn5ZdfZseOHWzfvp2XXnqJ/v37\nV/q9NW/enJYtW/LOO+8A8MwzzzBgwACKiopYtWoVAwcOZPz48WzZsoVt27axfPlyTj75ZG677Ta+\n8Y1vKOlL2qWzeSnR3kOp7n1UYfOOuxea2Q3AdKA+MNndF5rZOMKobtOA+4AjgP8zM4DPo5463YHH\nzKyIcIC5x91rZdKH0MRz4YUXHtSTZ8SIEZx//vmcfPLJ5OTk0K1bt3K3cd1113HFFVfQvXt3unfv\nvv8Xw6mnnkqvXr3o1q0bHTp0OGhY5lGjRjFo0KD9bfvFevfuzY9+9CP69u0LhBO5vXr1KrcppyxP\nPfXU/hO5Xbp04YknnmDfvn1cdtllbNmyBXfnxhtvpEWLFvziF79gxowZ1KtXjx49euy/C5hIbZRo\n81KiYxeleuwjK++kYzrk5OR4yX7rixcvpnv37mmKSKpCn5lkinr1Qg29JLPwy6O61z9Q3ua4e05F\n5XRFrohIAhI9J5Dq+3or6YuIJCDRcwKpvq93rUn6Na0ZSsqmz0oySaK9h1J9X+9a0ab/6aef0qxZ\nM1q1akV0olhqKHdn06ZNfP3113Tu3Dnd4YhkjHjb9GvFePrt27enoKCADRs2pDsUiUPjxo11wZZI\nDVUrkn7Dhg1VaxQRSYJa06YvIiKJU9IXEckgSvoiIhmkxvXeMbMNQPkD2JSvNbAxSeFUB8WXGMWX\nGMWXmJocXyd3r3DEyhqX9BNlZvnxdFtKF8WXGMWXGMWXmJoeXzzUvCMikkGU9EVEMkhdTPqT0h1A\nBRRfYhRfYhRfYmp6fBWqc236IiJStrpY0xcRkTIo6YuIZJBamfTNbJCZLTWzZWY2upTlh5nZ89Hy\nWWaWlcLYOpjZDDNbZGYLzeymUsqcYWZbzGxe9PhlquKLiWGlmS2IXj+/lOVmZg9F+3C+mfVOYWwn\nxuybeWa21cxuLlEmpfvQzCab2Xoz+yhm3lFm9rqZfRL9bVnGuiOjMp+Y2cgUxnefmS2JPr+XzKxF\nGeuW+12oxvjuMLPVMZ/hOWWsW+7/ezXG93xMbCvNbF4Z61b7/ksqd69VD8J9epcDXYBGwIdAdoky\nPwYejZ4PB55PYXzHAr2j582Aj0uJ7wzglTTvx5VA63KWnwO8ChjwLWBWGj/vLwgXnqRtHwKnA72B\nj2Lm3QuMjp6PBsaXst5RwIrob8voecsUxfddoEH0fHxp8cXzXajG+O4AfhbH51/u/3t1xVdi+QPA\nL9O1/5L5qI01/b7AMndf4e57gDxgaIkyQ4GnoudTgbMsRQPxu/tad58bPf8aWAy0S8VrJ9lQ4GkP\n3gNamNmxaYjjLGC5uydylXbC3H0m8GWJ2bHfs6eAC0pZ9XvA6+7+pbtvBl4HBqUiPnf/u7sXRpPv\nAWkb77qM/RePeP7fE1ZefFHuuBh4Ltmvmw61Mem3A1bFTBdwaFLdXyb60m8BWqUkuhhRs1IvYFYp\ni79tZh+a2atm1iOlgQUO/N3M5pjZqFKWx7OfU2E4Zf+zpXsfHuPua6PnXwDHlFKmpuzHKwm/3EpT\n0XehOt0QNT9NLqN5rCbsv/7AOnf/pIzl6dx/lVYbk36tYGZHAC8CN7v71hKL5xKaK04FHgZeTnV8\nwGnu3hsYDFxvZqenIYZymVkjYAjwf6Usrgn7cD8Pv/NrZP9nMxsDFAK5ZRRJ13fhEeB4oCewltCE\nUhNdQvm1/Br/vxSrNib91UCHmOn20bxSy5hZA6A5sCkl0YXXbEhI+Lnu/qeSy919q7tvi57/DWho\nZq1TFV/0uqujv+uBlwg/o2PFs5+r22BgrruvK7mgJuxDYF1xk1f0d30pZdK6H83sR8B5wIjowHSI\nOL4L1cLd17n7PncvAv5Qxuume/81AC4Cni+rTLr2X1XVxqQ/G+hqZp2jmuBwYFqJMtOA4l4Sw4C3\nyvrCJ1vU/vdHYLG7TyijTNvicwxm1pfwOaTyoNTUzJoVPyec8PuoRLFpwOVRL55vAVtimjJSpcwa\nVrr3YST2ezYS+HMpZaYD3zWzllHzxXejedXOzAYB/wMMcfcdZZSJ57tQXfHFniO6sIzXjef/vTqd\nDSxx94LSFqZz/1VZus8kV+VB6FnyMeGs/pho3jjClxugMaFJYBnwPtAlhbGdRviZPx+YFz3OAa4F\nro3K3AAsJPREeA/4zxTvvy7Ra38YxVG8D2NjNGBitI8XADkpjrEpIYk3j5mXtn1IOPisBfYS2pWv\nIpwnehP4BHgDOCoqmwM8HrPuldF3cRlwRQrjW0ZoDy/+Hhb3aDsO+Ft534UUxfdM9N2aT0jkx5aM\nL5o+5P89FfFF858s/s7FlE35/kvmQ8MwiIhkkNrYvCMiIlWkpC8ikkGU9EVEMoiSvohIBlHSFxHJ\nIEr6IiIZRElfRCSD/H9HJBKfh9lz3gAAAABJRU5ErkJggg==\n",
|
|
"text/plain": [
|
|
"<matplotlib.figure.Figure at 0x7f0637d633c8>"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
}
|
|
],
|
|
"source": [
|
|
"import matplotlib.pyplot as plt\n",
|
|
"\n",
|
|
"loss = history.history['loss']\n",
|
|
"val_loss = history.history['val_loss']\n",
|
|
"\n",
|
|
"epochs = range(len(loss))\n",
|
|
"\n",
|
|
"plt.figure()\n",
|
|
"\n",
|
|
"plt.plot(epochs, loss, 'bo', label='Training loss')\n",
|
|
"plt.plot(epochs, val_loss, 'b', label='Validation loss')\n",
|
|
"plt.title('Training and validation loss')\n",
|
|
"plt.legend()\n",
|
|
"\n",
|
|
"plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"\n",
|
|
"The validation MAE stays in the low 0.40s: we cannot even beat our common-sense baseline using the small convnet. Again, this is because \n",
|
|
"our convnet looks for patterns anywhere in the input timeseries, and has no knowledge of the temporal position of a pattern it sees (e.g. \n",
|
|
"towards the beginning, towards the end, etc.). Since more recent datapoints should be interpreted differently from older datapoints in the \n",
|
|
"case of this specific forecasting problem, the convnet fails at producing meaningful results here. This limitation of convnets was not an \n",
|
|
"issue on IMDB, because patterns of keywords that are associated with a positive or a negative sentiment will be informative independently \n",
|
|
"of where they are found in the input sentences.\n",
|
|
"\n",
|
|
"One strategy to combine the speed and lightness of convnets with the order-sensitivity of RNNs is to use a 1D convnet as a preprocessing \n",
|
|
"step before a RNN. This is especially beneficial when dealing with sequences that are so long that they couldn't realistically be processed \n",
|
|
"with RNNs, e.g. sequences with thousands of steps. The convnet will turn the long input sequence into much shorter (downsampled) sequences \n",
|
|
"of higher-level features. This sequence of extracted features then becomes the input to the RNN part of the network."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"\n",
|
|
"This technique is not seen very often in research papers and practical applications, possibly because it is not very well known. It is very \n",
|
|
"effective and ought to be more common. Let's try this out on the temperature forecasting dataset. Because this strategy allows us to \n",
|
|
"manipulate much longer sequences, we could either look at data from further back (by increasing the `lookback` parameter of the data \n",
|
|
"generator), or look at high-resolution timeseries (by decreasing the `step` parameter of the generator). Here, we will chose (somewhat \n",
|
|
"arbitrarily) to use a `step` twice smaller, resulting in twice longer timeseries, where the weather data is being sampled at a rate of one \n",
|
|
"point per 30 minutes."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 11,
|
|
"metadata": {
|
|
"collapsed": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# This was previously set to 6 (one point per hour).\n",
|
|
"# Now 3 (one point per 30 min).\n",
|
|
"step = 3\n",
|
|
"lookback = 720 # Unchanged\n",
|
|
"delay = 144 # Unchanged\n",
|
|
"\n",
|
|
"train_gen = generator(float_data,\n",
|
|
" lookback=lookback,\n",
|
|
" delay=delay,\n",
|
|
" min_index=0,\n",
|
|
" max_index=200000,\n",
|
|
" shuffle=True,\n",
|
|
" step=step)\n",
|
|
"val_gen = generator(float_data,\n",
|
|
" lookback=lookback,\n",
|
|
" delay=delay,\n",
|
|
" min_index=200001,\n",
|
|
" max_index=300000,\n",
|
|
" step=step)\n",
|
|
"test_gen = generator(float_data,\n",
|
|
" lookback=lookback,\n",
|
|
" delay=delay,\n",
|
|
" min_index=300001,\n",
|
|
" max_index=None,\n",
|
|
" step=step)\n",
|
|
"val_steps = (300000 - 200001 - lookback) // 128\n",
|
|
"test_steps = (len(float_data) - 300001 - lookback) // 128"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"This is our model, starting with two `Conv1D` layers and following-up with a `GRU` layer:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 12,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"_________________________________________________________________\n",
|
|
"Layer (type) Output Shape Param # \n",
|
|
"=================================================================\n",
|
|
"conv1d_6 (Conv1D) (None, None, 32) 2272 \n",
|
|
"_________________________________________________________________\n",
|
|
"max_pooling1d_4 (MaxPooling1 (None, None, 32) 0 \n",
|
|
"_________________________________________________________________\n",
|
|
"conv1d_7 (Conv1D) (None, None, 32) 5152 \n",
|
|
"_________________________________________________________________\n",
|
|
"gru_1 (GRU) (None, 32) 6240 \n",
|
|
"_________________________________________________________________\n",
|
|
"dense_3 (Dense) (None, 1) 33 \n",
|
|
"=================================================================\n",
|
|
"Total params: 13,697\n",
|
|
"Trainable params: 13,697\n",
|
|
"Non-trainable params: 0\n",
|
|
"_________________________________________________________________\n",
|
|
"Epoch 1/20\n",
|
|
"500/500 [==============================] - 60s - loss: 0.3387 - val_loss: 0.3030\n",
|
|
"Epoch 2/20\n",
|
|
"500/500 [==============================] - 58s - loss: 0.3055 - val_loss: 0.2864\n",
|
|
"Epoch 3/20\n",
|
|
"500/500 [==============================] - 58s - loss: 0.2904 - val_loss: 0.2841\n",
|
|
"Epoch 4/20\n",
|
|
"500/500 [==============================] - 58s - loss: 0.2830 - val_loss: 0.2730\n",
|
|
"Epoch 5/20\n",
|
|
"500/500 [==============================] - 58s - loss: 0.2767 - val_loss: 0.2757\n",
|
|
"Epoch 6/20\n",
|
|
"500/500 [==============================] - 58s - loss: 0.2696 - val_loss: 0.2819\n",
|
|
"Epoch 7/20\n",
|
|
"500/500 [==============================] - 57s - loss: 0.2642 - val_loss: 0.2787\n",
|
|
"Epoch 8/20\n",
|
|
"500/500 [==============================] - 57s - loss: 0.2595 - val_loss: 0.2920\n",
|
|
"Epoch 9/20\n",
|
|
"500/500 [==============================] - 58s - loss: 0.2546 - val_loss: 0.2919\n",
|
|
"Epoch 10/20\n",
|
|
"500/500 [==============================] - 57s - loss: 0.2506 - val_loss: 0.2772\n",
|
|
"Epoch 11/20\n",
|
|
"500/500 [==============================] - 58s - loss: 0.2459 - val_loss: 0.2801\n",
|
|
"Epoch 12/20\n",
|
|
"500/500 [==============================] - 57s - loss: 0.2433 - val_loss: 0.2807\n",
|
|
"Epoch 13/20\n",
|
|
"500/500 [==============================] - 58s - loss: 0.2411 - val_loss: 0.2855\n",
|
|
"Epoch 14/20\n",
|
|
"500/500 [==============================] - 58s - loss: 0.2360 - val_loss: 0.2858\n",
|
|
"Epoch 15/20\n",
|
|
"500/500 [==============================] - 57s - loss: 0.2350 - val_loss: 0.2834\n",
|
|
"Epoch 16/20\n",
|
|
"500/500 [==============================] - 58s - loss: 0.2315 - val_loss: 0.2917\n",
|
|
"Epoch 17/20\n",
|
|
"500/500 [==============================] - 60s - loss: 0.2285 - val_loss: 0.2944\n",
|
|
"Epoch 18/20\n",
|
|
"500/500 [==============================] - 57s - loss: 0.2280 - val_loss: 0.2923\n",
|
|
"Epoch 19/20\n",
|
|
"500/500 [==============================] - 57s - loss: 0.2249 - val_loss: 0.2910\n",
|
|
"Epoch 20/20\n",
|
|
"500/500 [==============================] - 57s - loss: 0.2215 - val_loss: 0.2952\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"model = Sequential()\n",
|
|
"model.add(layers.Conv1D(32, 5, activation='relu',\n",
|
|
" input_shape=(None, float_data.shape[-1])))\n",
|
|
"model.add(layers.MaxPooling1D(3))\n",
|
|
"model.add(layers.Conv1D(32, 5, activation='relu'))\n",
|
|
"model.add(layers.GRU(32, dropout=0.1, recurrent_dropout=0.5))\n",
|
|
"model.add(layers.Dense(1))\n",
|
|
"\n",
|
|
"model.summary()\n",
|
|
"\n",
|
|
"model.compile(optimizer=RMSprop(), loss='mae')\n",
|
|
"history = model.fit_generator(train_gen,\n",
|
|
" steps_per_epoch=500,\n",
|
|
" epochs=20,\n",
|
|
" validation_data=val_gen,\n",
|
|
" validation_steps=val_steps)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 13,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"<matplotlib.figure.Figure at 0x7f707ade8128>"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEICAYAAACzliQjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmYFNW5x/Hvyy6LgIhxQRhwBWQfQYMEUKO4gRiugkPc\ngxKJGq4mRNQYlBvFDTFc16txQdFoVFSUGCUxJnEBRBSQgAg6grIoII6Kw7z3j1MzNMMsPdM93dPT\nv8/z9DNd1aeq3q7ueav61KlzzN0REZHsUC/dAYiISOoo6YuIZBElfRGRLKKkLyKSRZT0RUSyiJK+\niEgWUdKXKjGz+ma21czaJ7NsOpnZgWaW9LbLZnasma2KmV5mZgPiKVuNbd1nZldWd/kK1nu9mf0x\n2euV9GmQ7gCkZpnZ1pjJpsB3wPZo+kJ3n1GV9bn7dqB5sstmA3c/JBnrMbMLgNHuPihm3RckY91S\n9ynp13HuXpJ0ozPJC9z9r+WVN7MG7l6YithEJPVUvZPlop/vj5vZY2b2FTDazI40szfMbJOZrTWz\naWbWMCrfwMzczHKi6Uei1180s6/M7N9m1rGqZaPXTzCz/5jZZjO7w8z+aWbnlBN3PDFeaGYrzOxL\nM5sWs2x9M7vNzDaa2UpgSAX7Z6KZzSw1b7qZ3Ro9v8DMlkbv58PoLLy8deWb2aDoeVMzeziKbTHQ\np1TZq8xsZbTexWY2NJrfDfgDMCCqOtsQs2+vjVn+oui9bzSzZ8xsn3j2TWXMbHgUzyYze9XMDol5\n7UozW2NmW8zsg5j3eoSZLYjmf25mN8W7PakB7q5HljyAVcCxpeZdD2wDTiGcBOwGHA70I/wS7AT8\nBxgXlW8AOJATTT8CbABygYbA48Aj1Si7F/AVMCx6bTzwPXBOOe8lnhifBVoCOcAXxe8dGAcsBtoB\nbYDXwr9CmdvpBGwFmsWsex2QG02fEpUx4GjgG6B79NqxwKqYdeUDg6LnNwN/A1oDHYAlpcqeDuwT\nfSZnRjH8IHrtAuBvpeJ8BLg2en5cFGNPoAnwv8Cr8eybMt7/9cAfo+edoziOjj6jK4Fl0fOuwGpg\n76hsR6BT9PxtYFT0vAXQL93/C9n80Jm+ALzu7s+5e5G7f+Pub7v7m+5e6O4rgXuAgRUs/6S7z3P3\n74EZhGRT1bInAwvd/dnotdsIB4gyxRnj7919s7uvIiTY4m2dDtzm7vnuvhG4oYLtrATeJxyMAH4M\nfOnu86LXn3P3lR68CrwClHmxtpTTgevd/Ut3X004e4/d7hPuvjb6TB4lHLBz41gvQB5wn7svdPdv\ngQnAQDNrF1OmvH1TkZHALHd/NfqMbiAcOPoBhYQDTNeoivCjaN9BOHgfZGZt3P0rd38zzvchNUBJ\nXwA+iZ0ws0PN7AUz+8zMtgCTgD0rWP6zmOcFVHzxtryy+8bG4e5OODMuU5wxxrUtwhlqRR4FRkXP\nz4ymi+M42czeNLMvzGwT4Sy7on1VbJ+KYjCzc8zs3agaZRNwaJzrhfD+Stbn7luAL4H9YspU5TMr\nb71FhM9oP3dfBvw34XNYF1UX7h0VPRfoAiwzs7fM7MQ434fUACV9gfBzP9bdhLPbA919d+AaQvVF\nTVpLqG4BwMyMnZNUaYnEuBbYP2a6sialTwDHmtl+hDP+R6MYdwOeBH5PqHppBfwlzjg+Ky8GM+sE\n3AmMBdpE6/0gZr2VNS9dQ6gyKl5fC0I10qdxxFWV9dYjfGafArj7I+7en1C1U5+wX3D3Ze4+klCF\ndwvwlJk1STAWqSYlfSlLC2Az8LWZdQYuTME2nwd6m9kpZtYAuBRoW0MxPgFcZmb7mVkb4NcVFXb3\nz4DXgT8Cy9x9efRSY6ARsB7YbmYnA8dUIYYrzayVhfsYxsW81pyQ2NcTjn8/I5zpF/scaFd84boM\njwHnm1l3M2tMSL7/cPdyfzlVIeahZjYo2vYVhOswb5pZZzMbHG3vm+hRRHgDPzWzPaNfBpuj91aU\nYCxSTUr6Upb/Bs4m/EPfTbjgWqPc/XPgDOBWYCNwAPAO4b6CZMd4J6Hu/T3CRcYn41jmUcKF2ZKq\nHXffBPwSeJpwMXQE4eAVj98SfnGsAl4EHopZ7yLgDuCtqMwhQGw9+MvAcuBzM4utpile/iVCNcvT\n0fLtCfX8CXH3xYR9fifhgDQEGBrV7zcGphCuw3xG+GUxMVr0RGCphdZhNwNnuPu2ROOR6rFQdSpS\nu5hZfUJ1wgh3/0e64xGpK3SmL7WGmQ2JqjsaA1cTWn28leawROoUJX2pTY4CVhKqDo4Hhrt7edU7\nIlINqt4REckiOtMXEckita7DtT333NNzcnLSHYaISEaZP3/+BnevqJkzUAuTfk5ODvPmzUt3GCIi\nGcXMKruzHFD1johIVlHSFxHJIkr6IiJZJK46fTMbAtxO6ETpPne/odTrFwEXE4bh2wqMcfclMa+3\nJ/QXfq2735yk2EUkCb7//nvy8/P59ttv0x2KxKFJkya0a9eOhg3L63qpYpUm/eh2+OmEfsTzgbfN\nbFZsUgcedfe7ovJDCf2nxI5GdCuhfxERqWXy8/Np0aIFOTk5hM5NpbZydzZu3Eh+fj4dO3asfIEy\nxFO90xdYEQ0UsQ2YyY4BJYoD2RIz2YyYrl/N7FTgI8JIRTVmxgzIyYF69cLfGVUa7lske3377be0\nadNGCT8DmBlt2rRJ6FdZPNU7+7HzYA/5hJFySgdzMWGIu0aE4dQws+aEbmt/DFxe3gbMbAwwBqB9\n+8q6Nt/VjBkwZgwUFITp1avDNEBewn0LitR9SviZI9HPKmkXct19ursfQEjyV0WzryUMS7e1kmXv\ncfdcd89t27bSewt2MXHijoRfrKAgzBcRkR3iSfqfsvMIPyUj5ZRjJnBq9LwfMMXMVgGXEQaNGFfe\ngtX18cdVmy8itcfGjRvp2bMnPXv2ZO+992a//fYrmd62Lb5u988991yWLVtWYZnp06czI0n1vkcd\ndRQLFy5MyrpSLZ7qnbcJgxp3JCT7kYRxQkuY2UExowmdRBjgAXcfEFPmWmCru+80AHQytG8fqnTK\nmi8iyTVjRvgV/fHH4X9s8uTEqlHbtGlTkkCvvfZamjdvzuWX71wb7O64O/XqlX2e+sADD1S6nYsv\nvrj6QdYhlZ7pu3shYSi3OcBS4Al3X2xmk6KWOgDjzGyxmS0k1OufXWMRl2HyZGjadOd5TZuG+SKS\nPMXXz1avBvcd189qouHEihUr6NKlC3l5eXTt2pW1a9cyZswYcnNz6dq1K5MmTSopW3zmXVhYSKtW\nrZgwYQI9evTgyCOPZN26dQBcddVVTJ06taT8hAkT6Nu3L4cccgj/+te/APj666/5yU9+QpcuXRgx\nYgS5ubmVntE/8sgjdOvWjcMOO4wrr7wSgMLCQn7605+WzJ82bRoAt912G126dKF79+6MHj066fss\nHnG103f32cDsUvOuiXl+aRzruLaqwcWr+CwjmWcfIrKriq6f1cT/2wcffMBDDz1Ebm4uADfccAN7\n7LEHhYWFDB48mBEjRtClS5edltm8eTMDBw7khhtuYPz48dx///1MmDBhl3W7O2+99RazZs1i0qRJ\nvPTSS9xxxx3svffePPXUU7z77rv07t27wvjy8/O56qqrmDdvHi1btuTYY4/l+eefp23btmzYsIH3\n3nsPgE2bNgEwZcoUVq9eTaNGjUrmpVqduSM3Lw9WrYKiovBXCV8k+VJ9/eyAAw4oSfgAjz32GL17\n96Z3794sXbqUJUuW7LLMbrvtxgknnABAnz59WLVqVZnrPu2003Yp8/rrrzNy5EgAevToQdeuXSuM\n78033+Too49mzz33pGHDhpx55pm89tprHHjggSxbtoxLLrmEOXPm0LJlSwC6du3K6NGjmTFjRrVv\nrkpUnUn6IlLzyrtOVlPXz5o1a1byfPny5dx+++28+uqrLFq0iCFDhpTZXr1Ro0Ylz+vXr09hYWGZ\n627cuHGlZaqrTZs2LFq0iAEDBjB9+nQuvPBCAObMmcNFF13E22+/Td++fdm+fXtStxsPJX0RiVs6\nr59t2bKFFi1asPvuu7N27VrmzJmT9G3079+fJ554AoD33nuvzF8Ssfr168fcuXPZuHEjhYWFzJw5\nk4EDB7J+/Xrcnf/6r/9i0qRJLFiwgO3bt5Ofn8/RRx/NlClT2LBhAwWl68pSoNb1py8itVc6r5/1\n7t2bLl26cOihh9KhQwf69++f9G384he/4KyzzqJLly4lj+KqmbK0a9eO6667jkGDBuHunHLKKZx0\n0kksWLCA888/H3fHzLjxxhspLCzkzDPP5KuvvqKoqIjLL7+cFi1aJP09VKbWjZGbm5vrGkRFJHWW\nLl1K586d0x1GrVBYWEhhYSFNmjRh+fLlHHfccSxfvpwGDWrX+XFZn5mZzXf33HIWKVG73omISBpt\n3bqVY445hsLCQtydu+++u9Yl/ETVrXcjIpKAVq1aMX/+/HSHUaN0IVdEJIso6YuIZBElfRGRLKKk\nLyKSRZT0RSStBg8evMuNVlOnTmXs2LEVLte8eXMA1qxZw4gRI8osM2jQICprAj516tSdbpI68cQT\nk9IvzrXXXsvNN9e+IcGV9EUkrUaNGsXMmTN3mjdz5kxGjRoV1/L77rsvTz75ZLW3Xzrpz549m1at\nWlV7fbWdkr6IpNWIESN44YUXSgZMWbVqFWvWrGHAgAEl7eZ79+5Nt27dePbZZ3dZftWqVRx22GEA\nfPPNN4wcOZLOnTszfPhwvvnmm5JyY8eOLemW+be//S0A06ZNY82aNQwePJjBgwcDkJOTw4YNGwC4\n9dZbOeywwzjssMNKumVetWoVnTt35mc/+xldu3bluOOO22k7ZVm4cCFHHHEE3bt3Z/jw4Xz55Zcl\n2y/uarm4o7e///3vJYPI9OrVi6+++qra+7YsaqcvIiUuuwySPSBUz54Q5csy7bHHHvTt25cXX3yR\nYcOGMXPmTE4//XTMjCZNmvD000+z++67s2HDBo444giGDh1a7jixd955J02bNmXp0qUsWrRop66R\nJ0+ezB577MH27ds55phjWLRoEZdccgm33norc+fOZc8999xpXfPnz+eBBx7gzTffxN3p168fAwcO\npHXr1ixfvpzHHnuMe++9l9NPP52nnnqqwv7xzzrrLO644w4GDhzINddcw+9+9zumTp3KDTfcwEcf\nfUTjxo1LqpRuvvlmpk+fTv/+/dm6dStNmjSpwt6unM70RSTtYqt4Yqt23J0rr7yS7t27c+yxx/Lp\np5/y+eefl7ue1157rST5du/ene7du5e89sQTT9C7d2969erF4sWLK+1M7fXXX2f48OE0a9aM5s2b\nc9ppp/GPf/wDgI4dO9KzZ0+g4u6bIfTvv2nTJgYOHAjA2WefzWuvvVYSY15eHo888kjJnb/9+/dn\n/PjxTJs2jU2bNiX9jmCd6YtIiYrOyGvSsGHD+OUvf8mCBQsoKCigT58+AMyYMYP169czf/58GjZs\nSE5OTpndKVfmo48+4uabb+btt9+mdevWnHPOOdVaT7HibpkhdM1cWfVOeV544QVee+01nnvuOSZP\nnsx7773HhAkTOOmkk5g9ezb9+/dnzpw5HHroodWOtTSd6YtI2jVv3pzBgwdz3nnn7XQBd/Pmzey1\n1140bNiQuXPnsrqswbBj/OhHP+LRRx8F4P3332fRokVA6Ja5WbNmtGzZks8//5wXX3yxZJkWLVqU\nWW8+YMAAnnnmGQoKCvj66695+umnGTBgwC7lKtOyZUtat25d8ivh4YcfZuDAgRQVFfHJJ58wePBg\nbrzxRjZv3szWrVv58MMP6datG7/+9a85/PDD+eCDD6q8zYroTF9EaoVRo0YxfPjwnVry5OXlccop\np9CtWzdyc3MrPeMdO3Ys5557Lp07d6Zz584lvxh69OhBr169OPTQQ9l///136pZ5zJgxDBkyhH33\n3Ze5c+eWzO/duzfnnHMOffv2BeCCCy6gV69eFVbllOfBBx/koosuoqCggE6dOvHAAw+wfft2Ro8e\nzebNm3F3LrnkElq1asXVV1/N3LlzqVevHl27di0ZBSxZ1LWySJZT18qZJ5GulVW9IyKSRZT0RUSy\niJK+iFDbqnmlfIl+Vkr6IlmuSZMmbNy4UYk/A7g7GzduTOiGLbXeEcly7dq1Iz8/n/Xr16c7FIlD\nkyZNaNeuXbWXjyvpm9kQ4HagPnCfu99Q6vWLgIuB7cBWYIy7LzGzHwM3AI2AbcAV7v5qtaMVkaRr\n2LAhHTt2THcYkiKVVu+YWX1gOnAC0AUYZWZdShV71N27uXtPYApwazR/A3CKu3cDzgYeTlrkIiJS\nZfHU6fcFVrj7SnffBswEhsUWcPctMZPNAI/mv+Pua6L5i4HdzKwxIiKSFvFU7+wHfBIznQ/0K13I\nzC4GxhOqco4uYz0/ARa4+3dlLDsGGAPQvn37OEISEZHqSFrrHXef7u4HAL8Grop9zcy6AjcCF5az\n7D3unuvuuW3btk1WSCIiUko8Sf9TYP+Y6XbRvPLMBE4tnjCzdsDTwFnu/mF1ghQRkeSIJ+m/DRxk\nZh3NrBEwEpgVW8DMDoqZPAlYHs1vBbwATHD3fyYnZBERqa5Kk767FwLjgDnAUuAJd19sZpPMbGhU\nbJyZLTazhYR6/bOL5wMHAteY2cLosVfy34aIiMRDvWyKiNQB6mVTRER2oaQvIpJFlPRFRLKIkr6I\nSBZR0hcRySJK+iIiWURJX0Qkiyjpi4hkESV9EZEsoqQvIpJFlPRFRLKIkr6ISBZR0hcRySJK+iIi\nWURJX0QkiyjpR2bMgJwcqFcv/J0xI90RiYgkX4N0B1AbzJgBY8ZAQUGYXr06TAPk5aUvLhGRZNOZ\nPjBx4o6EX6ygIMwXEalLlPSBjz+u2nwRkUylpA+0b1+1+SIimUpJH5g8GZo23Xle06ZhvohIXVJn\nkr47PP88bNpU9WXz8uCee6BDBzALf++5RxdxRaTuqTNJf/lyGDoUrr++esvn5cGqVVBUFP4q4YtI\nXVRnkv7BB8N558G0aeEAICIiu4or6ZvZEDNbZmYrzGxCGa9fZGbvmdlCM3vdzLrEvPabaLllZnZ8\nMoMvbfJkaNIE/vu/a3IrIiKZq9Kkb2b1genACUAXYFRsUo886u7d3L0nMAW4NVq2CzAS6AoMAf43\nWl+N+MEP4Kqr4Lnn4OWXa2orIiKZK54z/b7ACndf6e7bgJnAsNgC7r4lZrIZ4NHzYcBMd//O3T8C\nVkTrqzGXXgoHHAC//CUUFtbklkREMk88SX8/4JOY6fxo3k7M7GIz+5Bwpn9JFZcdY2bzzGze+vXr\n4429TI0bw803w+LFcPfdCa1KRKTOSdqFXHef7u4HAL8Grqrisve4e66757Zt2zbhWIYNg8GD4Zpr\n4IsvEl6diEidEU/S/xTYP2a6XTSvPDOBU6u5bFKYwdSpoc3+735X01sTEckc8ST9t4GDzKyjmTUi\nXJidFVvAzA6KmTwJKG40OQsYaWaNzawjcBDwVuJhV657d/jZz2D6dFi6NBVbFBGp/SpN+u5eCIwD\n5gBLgSfcfbGZTTKzoVGxcWa22MwWAuOBs6NlFwNPAEuAl4CL3X17DbyPMl13HTRvDuPHp2qLIiK1\nm7l75aVSKDc31+fNm5e09d16a2i3/8ILcOKJSVutiEitYmbz3T23snJ15o7c8owbF+7WHT8evv8+\n3dGIiKRXnU/6jRrBLbfAsmWhfl9EpDb66qvUXH+s80kf4KST4LjjQkueDRvSHY2IyA4rV4abSdu1\ngzPPDD0G16SsSPpmcNtt4Uh6zTXpjkZEkmX7dnjzTfjtb+GII6Bv33BX/uOPwyefVL58urjD3/4G\np54KBx4If/gDnHxyuKHUrGa3Xecv5Mb6xS/gf/8XFi6Ebt1qZBMiUsM2boQ5c2D27PB3w4aQKPv1\nCx0uvvXWjjGv27WDH/5wx6NnT2jYMH2xf/stPPYY3H47vPsutGkDF10EY8fCfrv0VVA18V7Izaqk\nv3EjHHQQ9O4dOmSr6SOqiCSuqAjeeSck+dmzQ1IvKoI994QhQ0KrvOOOCwkUQoONRYvgX//a8Sge\n73q33eDww3ccBI48Mqynpn32Gdx5Z3isXw+HHRZ+keTlhZiSQUm/HHfcAZdcAs88E7prEJHa58sv\nw4nZ7Nnw4ouwbl04ScvNDUn+xBOhTx+oH2efvfn58O9/7zgILFiwo0PGgw8Oyf+HP4RevaBjx3AA\nScZJ4fz54ax+5sywvZNPDsn+6KOTf9KppF+O77+HHj1g27bQKVvjxjW2KamCoqLQLfaiRWG6+B8i\n9m9Z82L/1q8fqvAGDEhNzJJcK1eGuvjZs0OC3r4dWreG448PSf7442GvvZKzrW++gXnzdv41ENvI\no0WLkPw7dQqP4ucdO0JOTsVn54WF8OyzoSuY118PN4iee274bh50UPnLJUpJvwJz5oSfhVOmwBVX\nJGedM2bAxInhZ2T79mFAFw25GL/im+i6ddtR51r81XTf+Xl5r332GdSrF5q9Ff/Ul8wwZw6MGAFb\nt4bq1xNPhBNOCBdmGzSo+e27w4oVsGQJfPRROADF/v3mm53L77PPrgeDjh3DgeSOO2D16nBwuOSS\nMKJfy5Y1/x7iTfq4e6169OnTx1PhpJPcW7Rw/+yzxNf1yCPuTZsWp5/waNo0zJfKvfOOe6NG7sOH\nuxcVVX89Cxe6N2jgftZZyYtNat5997nXr+/eo4f7ypXpjmZXRUXua9e6/+tf4X/6uuvczz3XfdAg\n9/bt3evV2/l/f+BA96efdi8sTG2cwDyPI8emPcmXfqQq6X/wQUgQF1yQ+Lo6dNj5Qy9+dOiQ+Lrr\nuq+/dj/0UPd993XfsCHx9U2cGPb9Sy8lvi6pWUVF7ldfHT6v449337Il3RFVz3ffuS9f7v6Xv7gv\nWpS+OOJN+llZvVNs/PhQ7zZ/friAU1316pV9Q4VZqKuW8o0dG9omv/wyHHNM4uv79tvQLO/bb+H9\n90N9qtQ+27bBBRfAww+H6o+77kpvU8q6QH3vxOGaa0Ld72WXJXYXXPv2VZsvwbPPhn/2yy9PTsKH\n0E77vvtCnepVVRrKR1Jl8+ZQX//wwzBpUvi8lPBTJ6uTfqtWofvl116Dp56q/nomT4amTXee17Rp\nmC9lW7MGzj8/XLS7/vrkrvuoo+DnP4dp00IrEKk9PvkkfD6vvQYPPghXX637ZVItq6t3IDSv6tUr\ntBpYujScKVaHWu/Er6goNL/75z9De+lDD03+NrZsga5dYffdwzbUNDf9Fi4M/WBt3Qp//nPyft1J\noOqdODVoEOr1V60KzQarKy8vrKOoKPxVwi/fbbfBX/8a9ntNJHwIyf6uu0ITvN//vma2IfGbMyfc\nP1GvXmi7roSfPlmf9CF8AYcNg//5n3CTRvGdepJ877wDv/kNDB8ehrOsSSedBKNGhc918eKa3ZaU\n7/77w2fRqRO88Yb6vUq3rK/eKbZixY5qnhYtQr3jwIHh0aePLjQlQ0FB2JdbtoQ7b1NxA9X69dC5\nc+jJ8J//jP+2/bqkqCjcXFRQEB7Fz8uat//+0L//rteoqsM99H553XWhb5w//Sn8ApOaEW/1Tgru\ndcsMBx4IH34Ir74Kf/97eLz4YnitWbPQL0fxQeDww1VHXB3jx4fBbF5+OXV3zLZtG/o+GT06dF97\n6aWp2W6qvftu+EWzdOmuyfy776q2roYNQ180Rx8dHv36hcGIqmLbtvBL7qGHQhcEd9+tE6faQmf6\nFVi3LrQyKD4IvPdemN+kSfinKD4I9OuXvJ7y6qpnnglVOldcEbq/SCX3UL3w97+Hap6cnNRuvya9\n/34YHOjJJ8Ot/gMHhpOUpk3Dd7Jp0/Kfl57XpEk4KL/6angsWBD2XdOmoT6++CDQq1fFv5g2b4af\n/AReeSXEphY6qaFuGGrAhg3h9urLLnPv1cvdLNxN2KiR+4AB4e7Czz+v3rofeSTcwWsW/talLhw+\n/dS9TRv33r3D3YvpsHq1e/Pm7scdl1hXD7XFkiXuZ5wRvi8tWoTv3hdfJHcbX3wRvu+/+IV71647\n7jRv2dJ92DD32293f++9nffnxx+7H3ZYuNv9j39MbjxSMdQNQ8378kv3555zv/xy98MPD31w9Orl\n/tVXVVtPXe67Z/t292OOCe/ngw/SG8sdd4R9++CD6Y0jEcuWueflhWTfrJn7lVcmp/uKeKxd6/7Y\nY6Hrkk6ddnxX99rLfeRI96lT3ffbLxyEXn45NTHJDkr6aTB7dkj8J59ctc6W6nLfPTfdFN7LPfek\nO5JwAPrhD91bt05OR3uptHx56EiuXr1wAP3Vr9zXrUtvTB995H7//e6jR4e+kyAk/XffTW9c2UpJ\nP02mTw979ZJL4l+muJqo9MOs5uJMhfnz3Rs2TLz3zGRasiRUx51+erojic/Kle7nnRd6oWzSxH38\n+Np5wCoqCgemTO00rS6IN+mrnX6S/fznYWT7adNCa5F41Ia+ez75BC68MLShX7gwsb6IAL7+Gs48\nMwx6ce+9tedCXufO4cLiE0/ArFnpjqZ8q1fDmDFhVKcZM2DcuNC3+y23wA9+kO7odmUWWsC1aJHu\nSKRS8RwZgCHAMmAFMKGM18cDS4BFwCtAh5jXpgCLgaXANKIWQ+U9Mv1M3z1U7QwdGn6KP/985eXT\nWae/fbv7XXeFetgmTcIZJbgffHC4OPj++9Vb75gx4ZfKK68kN95k+O47927dQpXEpk3pjmZnn3zi\nPnZs+IXUqJH7xRe75+enOyrJBCSregeoD3wIdAIaAe8CXUqVGQw0jZ6PBR6Pnv8Q+Ge0jvrAv4FB\nFW2vLiR9d/etW0NrlWbNwiAhlUlH650VK8JAEBAutn74Yagnvusu98GDdwwO0bWr+6RJ4SJiPP78\n57Dcr35Vs/En4q23wvu78MKa20ZRkXtBQaiOWbbM/e233f/617B//vjH0Prluuvcr7giHCRPOy0k\n+oYN3S+6KLSEEYlXMpP+kcCcmOnfAL+poHwv4J8xy84HdgOaAvOAzhVtr64kfffQVLFdu3Bxqzad\nrRUWut99tOiNAAARBUlEQVRyi/tuu7nvvrv7vfeWXee+dm1o8XLUUTt+gfTs6f7734cDRFny8933\n2MO9T5/0Nc+M1/jx4T397W+Jreezz8LoT6eeGkZ/6tgxNFFt0KDsazWlH02ahBYwBx7o/rOfhQuk\nIlUVb9Kv9OYsMxsBDHH3C6LpnwL93H1cOeX/AHzm7tdH0zcDFwAG/MHdJ5axzBhgDED79u37rF69\nusKYMsmiReG29oMOCjd6pXtQj8WLQ5fGb74Jp5wCd94J++1X+XKffhpuo3/88dB/CoQ7k884A04/\nPdy+X1QUbrf/97/DjT2HHFKz7yVRX38d+oFp0CDc0VqVG+w++CCMBzBrVni/7tChA/ToEboa2H33\ncLNUWc9LT1f1bleRsiTt5ixgBHBfzPRPCcm7rLKjgTeAxtH0gcALQPPo8W9gQEXbq0tn+sVeeCFU\nJZxySurHzSy2bVuoomnY0H3PPd0ffbT6LWpWrXKfMiWczRefrR55pPuZZ4bn996b3Nhr0ssvh5gn\nTKi4XGGh++uvh6qYgw/e8b779HH/3e/C+Ly1pYWSZCdSXb0DHEu4WLtXzLwrgKtjpq8BflXR9upi\n0nd3/8Mfwt6+9NLUb3vePPfu3cP2R45Mbvvu5cvdJ08O1Rrg/pOfZF7yK24SuWDBzvO//tr92WfD\n623bhvfXsKH7j38cPk/VuUttksyk3wBYCXRkx4XcrqXK9CJc7D2o1PwzgL9G62hIaNlzSkXbq6tJ\n3z103wAhYSRbWReCCwrcf/3rkND22ScksJq0alXtr8cvyxdfuO+9d7ibes0a9//7v9DNwG67hc9r\n993DwfKxx2pfax+RYklL+mFdnAj8J0rsE6N5k4Ch0fO/Ap8DC6PHrGh+feDu6BfAEuDWyrZVl5N+\nbFPOF15I3nrLavLZuHFIZOB+/vmhywgp35NP7rz/9t/ffdy4UP2TiQcyyT7xJn31spliW7fCj34E\ny5eHEYR69Eh8nTk54Wae0urXh5degmOPTXwb2eB//id0CTxsGPTsWXtuKBOJR7wXcpX002DNmtAd\nM4RWNPvum9j66tUr/w7aWvbxikgN0Ri5tdi++8Lzz8OmTXDyyeHsvzry8+G++8pvatihQ/VjFJG6\nSUk/TXr0CG3e33039FGzfXvly2zbFga3+NWvQvvy/fcPoxM1abLroBZNm8LkyTUTu4hkLiX9NDrx\nxDCU33PPweWXl11m9Wq46y449dQwxOAxx8DUqaEjs5tuCiMnbdgADz4YzuzNwt977oG8vNS+HxGp\n/TRGbpqNGxcGZZ86NfRSeMEF4c7dF18MF2GXLg3lOnQI47yecEIYsq70nb15eUryIlI5XcitBbZv\nD+PHvvBCqKopKAgDrw8cCEOGhER/yCFqTSIi5Yv3Qq7O9GuB+vXh0UdDf/Z77BES/aBBYYBrEZFk\nUtKvJZo3D4NliIjUJF3IFRHJIkr6AoRfGTk54UavnBz96hCpq1S9I8yYEcZjLSgI08Xjs4JaBInU\nNTrTFyZO3JHwixUUhPkiUrco6Qsff1y1+SKSuZT0hfbtqzZfRDKXkr4weXLoqyeW+u4RqZuU9IW8\nvNBXj/ruEan71HpHAPXdI5ItdKYvIpJFlPQlKXRzl0hmUPWOJEw3d4lkDp3pS8J0c5dI5lDSl4Tp\n5i6RzKGkLwnTzV0imUNJXxKmm7tEMoeSviRMN3eJZI64kr6ZDTGzZWa2wswmlPH6eDNbYmaLzOwV\nM+sQ81p7M/uLmS2NyuQkL3ypLfLyYNUqKCoKf5XwRWqnSpO+mdUHpgMnAF2AUWbWpVSxd4Bcd+8O\nPAlMiXntIeAmd+8M9AXWJSNwqVvUzl8kNeI50+8LrHD3le6+DZgJDIst4O5z3b240d4bQDuA6ODQ\nwN1fjsptjSknAuxo5796NbjvaOevxC+SfPEk/f2AT2Km86N55TkfeDF6fjCwycz+bGbvmNlN0S+H\nnZjZGDObZ2bz1q9fH2/sUkeonb9I6iT1Qq6ZjQZygZuiWQ2AAcDlwOFAJ+Cc0su5+z3unuvuuW3b\ntk1mSJIB1M5fJHXiSfqfAvvHTLeL5u3EzI4FJgJD3f27aHY+sDCqGioEngF6Jxay1DVq5y+SOvEk\n/beBg8yso5k1AkYCs2ILmFkv4G5Cwl9XatlWZlZ8+n40sCTxsKUuUTt/kdSpNOlHZ+jjgDnAUuAJ\nd19sZpPMbGhU7CagOfAnM1toZrOiZbcTqnZeMbP3AAPurYH3IRlM7fxFUsfcPd0x7CQ3N9fnzZuX\n7jBERDKKmc1399zKyumOXKkT1M5fJD7qT18ynvrzF4mfzvQl46mdv0j8lPQl46mdv0j8lPQl46md\nv0j8lPQl46mdv0j8lPQl46mdv0j8lPSlTki0P381+ZRsoSabkvXU5FOyic70JeupyadkEyV9yXpq\n8inZRElfsp6afEo2UdKXrKcmn5JNlPQl66nJp2QTtd4RISR4JXnJBjrTF0kCtfOXTKEzfZEEqZ2/\nZBKd6YskSO38JZMo6YskSO38JZMo6YskKBnt/HVNQFJFSV8kQYm28y++JrB6NbjvuCagxC81QUlf\nJEGJtvPXNQFJJXP3dMewk9zcXJ83b166wxBJmXr1whl+aWahq2iReJjZfHfPrayczvRF0kzXBCSV\nlPRF0kzXBCSV4kr6ZjbEzJaZ2Qozm1DG6+PNbImZLTKzV8ysQ6nXdzezfDP7Q7ICF6krdE1AUqnS\nOn0zqw/8B/gxkA+8DYxy9yUxZQYDb7p7gZmNBQa5+xkxr98OtAW+cPdxFW1PdfoiVaNrAgLJrdPv\nC6xw95Xuvg2YCQyLLeDuc929+FzjDaBdTCB9gB8Af4k3eBGJn8YDkKqIJ+nvB3wSM50fzSvP+cCL\nAGZWD7gFuLyiDZjZGDObZ2bz1q9fH0dIIlJM4wFIVST1Qq6ZjQZygZuiWT8HZrt7fkXLufs97p7r\n7rlt27ZNZkgidZ7GA5CqiKeXzU+B/WOm20XzdmJmxwITgYHu/l00+0hggJn9HGgONDKzre6+y8Vg\nEak+jQcg8YrnTP9t4CAz62hmjYCRwKzYAmbWC7gbGOru64rnu3ueu7d39xxCFc9DSvgitY/a+WeP\nSs/03b3QzMYBc4D6wP3uvtjMJgHz3H0WoTqnOfAnMwP42N2H1mDcIpIkGg8gu6gbBpEsl5MTEn1p\nHTrAqlXxrWPGjHBfwMcfh1ZDkyfrgJFq8TbZ1MhZIlku0fEA9Eshs6gbBpEsl2g7f90RnFmU9EWy\nXKLt/DVyWGZR0hfJcom289cdwZlFSV9EyMsLF22LisLfqtTF647gzKKkLyIJScYdwbpPIHXUekdE\nEpbIHcFq/ZNaOtMXkbRS65/UUtIXkbRS65/UUtIXkbRS65/UUtIXkbRKRusfXQiOn5K+iKRVoq1/\nkjEwfDYdNNThmohktEQ7jCvdegjCL41MG4gmmWPkiojUWoleCM621kNK+iKS0RK9EJxtrYeU9EUk\noyV6ITjbWg8p6YtIRkv0QnC2tR5SNwwikvES6QaieLnqjvyVad1IqPWOiEgCkjHcZDKo9Y6ISApk\n2oVgJX0RkQRk2oVgJX0RkQRk2iAySvoiIgnItEFk1HpHRCRBmTSITFxn+mY2xMyWmdkKM5tQxuvj\nzWyJmS0ys1fMrEM0v6eZ/dvMFkevnZHsNyAikslS3Q1EpUnfzOoD04ETgC7AKDPrUqrYO0Cuu3cH\nngSmRPMLgLPcvSswBJhqZq2SFbyISKZLdeufeM70+wIr3H2lu28DZgLDYgu4+1x3Lz5WvQG0i+b/\nx92XR8/XAOuAtskKXkQk06W69U88SX8/4JOY6fxoXnnOB14sPdPM+gKNgA/LeG2Mmc0zs3nr16+P\nIyQRkboh1a1/ktp6x8xGA7nATaXm7wM8DJzr7kWll3P3e9w9191z27bVDwERyR7JaP1TFfG03vkU\n2D9mul00bydmdiwwERjo7t/FzN8deAGY6O5vJBauiEjdk0jrn6qK50z/beAgM+toZo2AkcCs2AJm\n1gu4Gxjq7uti5jcCngYecvcnkxe2iIhUR6VJ390LgXHAHGAp8IS7LzazSWY2NCp2E9Ac+JOZLTSz\n4oPC6cCPgHOi+QvNrGfy34aIiMRDvWyKiNQB6mVTRER2oaQvIpJFal31jpmtB8oYkiBuewIbkhRO\nTVB8iVF8iVF8ianN8XVw90rbvNe6pJ8oM5sXT71Wuii+xCi+xCi+xNT2+OKh6h0RkSyipC8ikkXq\nYtK/J90BVELxJUbxJUbxJaa2x1epOlenLyIi5auLZ/oiIlIOJX0RkSySkUk/juEbG5vZ49Hrb5pZ\nTgpj29/M5kbDRy42s0vLKDPIzDbH9Ed0Tarii4lhlZm9F21/l34vLJgW7cNFZtY7hbEdErNvFprZ\nFjO7rFSZlO5DM7vfzNaZ2fsx8/Yws5fNbHn0t3U5y54dlVluZmenML6bzOyD6PN7urxR6yr7LtRg\nfNea2acxn+GJ5Sxb4f97Dcb3eExsq8xsYTnL1vj+Syp3z6gHUJ8wEEsnwqAs7wJdSpX5OXBX9Hwk\n8HgK49sH6B09bwH8p4z4BgHPp3k/rgL2rOD1EwmD4RhwBPBmGj/vzwg3nqRtHxI6DuwNvB8zbwow\nIXo+AbixjOX2AFZGf1tHz1unKL7jgAbR8xvLii+e70INxnctcHkcn3+F/+81FV+p128BrknX/kvm\nIxPP9CsdvjGafjB6/iRwjJlZKoJz97XuviB6/hWhZ9KKRhqrrYYRusR2D+MgtIoGw0m1Y4AP3T2R\nu7QT5u6vAV+Umh37PXsQOLWMRY8HXnb3L9z9S+BlwnjRNR6fu//FQy+5EDOMaTqUs//iEc//e8Iq\nii/KHacDjyV7u+mQiUk/nuEbS8pEX/rNQJuURBcjqlbqBbxZxstHmtm7ZvaimXVNaWCBA38xs/lm\nNqaM16s6TGZNGUn5/2zp3oc/cPe10fPPgB+UUaa27MfzKGMY00hl34WaNC6qfrq/nOqx2rD/BgCf\nezTedxnSuf+qLBOTfkYws+bAU8Bl7r6l1MsLCNUVPYA7gGdSHR9wlLv3Bk4ALjazH6UhhgpZGIRn\nKPCnMl6uDfuwhIff+bWy/bOZTQQKgRnlFEnXd+FO4ACgJ7CWUIVSG42i4rP8Wv+/FCsTk348wzeW\nlDGzBkBLYGNKogvbbEhI+DPc/c+lX3f3Le6+NXo+G2hoZnumKr5ou59Gf9cRRjfrW6pIXMNk1rAT\ngAXu/nnpF2rDPgQ+L67yiv6uK6NMWvejmZ0DnAzkRQemXcTxXagR7v65u2/3MG72veVsN937rwFw\nGvB4eWXStf+qKxOTfqXDN0bTxa0kRgCvlveFT7ao/u//gKXufms5ZfYuvsZgZn0Jn0MqD0rNzKxF\n8XPCBb/3SxWbBZwVteI5AtgcU5WRKuWeYaV7H0Ziv2dnA8+WUWYOcJyZtY6qL46L5tU4MxsC/Iow\njGlBOWXi+S7UVHyx14iGl7PdeP7fa9KxwAfunl/Wi+ncf9WW7ivJ1XkQWpb8h3BVf2I0bxLhyw3Q\nhFAlsAJ4C+iUwtiOIvzMXwQsjB4nAhcBF0VlxgGLCS0R3gB+mOL91yna9rtRHMX7MDZGA6ZH+/g9\nIDfFMTYjJPGWMfPStg8JB5+1wPeEeuXzCdeJXgGWA38F9ojK5gL3xSx7XvRdXAGcm8L4VhDqw4u/\nh8Ut2vYFZlf0XUhRfA9H361FhES+T+n4ould/t9TEV80/4/F37mYsinff8l8qBsGEZEskonVOyIi\nUk1K+iIiWURJX0Qkiyjpi4hkESV9EZEsoqQvIpJFlPRFRLLI/wO0gNF2vd8PIQAAAABJRU5ErkJg\ngg==\n",
|
|
"text/plain": [
|
|
"<matplotlib.figure.Figure at 0x7f703c7fe940>"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
}
|
|
],
|
|
"source": [
|
|
"loss = history.history['loss']\n",
|
|
"val_loss = history.history['val_loss']\n",
|
|
"\n",
|
|
"epochs = range(len(loss))\n",
|
|
"\n",
|
|
"plt.figure()\n",
|
|
"\n",
|
|
"plt.plot(epochs, loss, 'bo', label='Training loss')\n",
|
|
"plt.plot(epochs, val_loss, 'b', label='Validation loss')\n",
|
|
"plt.title('Training and validation loss')\n",
|
|
"plt.legend()\n",
|
|
"\n",
|
|
"plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Judging from the validation loss, this setup is not quite as good as the regularized GRU alone, but it's significantly faster. It is \n",
|
|
"looking at twice more data, which in this case doesn't appear to be hugely helpful, but may be important for other datasets."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Wrapping up\n",
|
|
"\n",
|
|
"Here's what you should take away from this section:\n",
|
|
"\n",
|
|
"* In the same way that 2D convnets perform well for processing visual patterns in 2D space, 1D convnets perform well for processing \n",
|
|
"temporal patterns. They offer a faster alternative to RNNs on some problems, in particular NLP tasks.\n",
|
|
"* Typically 1D convnets are structured much like their 2D equivalents from the world of computer vision: they consist of stacks of `Conv1D` \n",
|
|
"layers and `MaxPooling1D` layers, eventually ending in a global pooling operation or flattening operation.\n",
|
|
"* Because RNNs are extremely expensive for processing very long sequences, but 1D convnets are cheap, it can be a good idea to use a 1D \n",
|
|
"convnet as a preprocessing step before a RNN, shortening the sequence and extracting useful representations for the RNN to process.\n",
|
|
"\n",
|
|
"One useful and important concept that we will not cover in these pages is that of 1D convolution with dilated kernels."
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.5.2"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 2
|
|
}
|