wehub-resource-sync
1107 lines
29 KiB
Plaintext
1107 lines
29 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "8849f2e0",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| hide\n",
|
|
"#| eval: false\n",
|
|
"! [ -e /content ] && pip install -Uqq fastai # upgrade fastai on colab"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "02949cab",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| default_exp callback.tensorboard"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "raw",
|
|
"id": "abcbde60",
|
|
"metadata": {},
|
|
"source": [
|
|
"---\n",
|
|
"skip_exec: true\n",
|
|
"---"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "26870397",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| export\n",
|
|
"from fastai.basics import *"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "0434b4c9",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from nbdev import show_doc"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "aa38e607",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Tensorboard\n",
|
|
"\n",
|
|
"> Integration with [tensorboard](https://www.tensorflow.org/tensorboard) "
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "390168a0",
|
|
"metadata": {},
|
|
"source": [
|
|
"First thing first, you need to install tensorboard with\n",
|
|
"```\n",
|
|
"pip install tensorboard\n",
|
|
"```\n",
|
|
"Then launch tensorboard with\n",
|
|
"``` \n",
|
|
"tensorboard --logdir=runs\n",
|
|
"```\n",
|
|
"in your terminal. You can change the logdir as long as it matches the `log_dir` you pass to `TensorBoardCallback` (default is `runs` in the working directory)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "c3f483ad",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Tensorboard Embedding Projector support"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "f027b225",
|
|
"metadata": {},
|
|
"source": [
|
|
"> Tensorboard Embedding Projector is currently only supported for image classification"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "cd5c3156",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Export Image Features during Training"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "2c7f4d56",
|
|
"metadata": {},
|
|
"source": [
|
|
"Tensorboard [Embedding Projector](https://www.tensorflow.org/tensorboard/tensorboard_projector_plugin) is supported in `TensorBoardCallback` (set parameter `projector=True`) during training. The validation set embeddings will be written after each epoch.\n",
|
|
"\n",
|
|
"```\n",
|
|
"cbs = [TensorBoardCallback(projector=True)]\n",
|
|
"learn = vision_learner(dls, resnet18, metrics=accuracy)\n",
|
|
"learn.fit_one_cycle(3, cbs=cbs)\n",
|
|
"```"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "600a772c",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Export Image Features during Inference"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "dbc91255",
|
|
"metadata": {},
|
|
"source": [
|
|
"To write the embeddings for a custom dataset (e. g. after loading a learner) use `TensorBoardProjectorCallback`. Add the callback manually to the learner.\n",
|
|
"\n",
|
|
"```\n",
|
|
"learn = load_learner('path/to/export.pkl')\n",
|
|
"learn.add_cb(TensorBoardProjectorCallback())\n",
|
|
"dl = learn.dls.test_dl(files, with_labels=True)\n",
|
|
"_ = learn.get_preds(dl=dl)\n",
|
|
"```"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "f4d761fc",
|
|
"metadata": {},
|
|
"source": [
|
|
"If using a custom model (non fastai-resnet) pass the layer where the embeddings should be extracted as a callback-parameter.\n",
|
|
"\n",
|
|
"```\n",
|
|
"layer = learn.model[1][1]\n",
|
|
"cbs = [TensorBoardProjectorCallback(layer=layer)]\n",
|
|
"preds = learn.get_preds(dl=dl, cbs=cbs)\n",
|
|
"```"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "67d07374",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Export Word Embeddings from Language Models"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "6a785c95",
|
|
"metadata": {},
|
|
"source": [
|
|
"To export word embeddings from Language Models (tested with AWD_LSTM (fast.ai) and GPT2 / BERT (transformers)) but works with every model that contains an embedding layer."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "6ea5e1ef",
|
|
"metadata": {},
|
|
"source": [
|
|
"For a **fast.ai TextLearner or LMLearner** just pass the learner - the embedding layer and vocab will be extracted automatically:\n",
|
|
"```\n",
|
|
"dls = TextDataLoaders.from_folder(untar_data(URLs.IMDB), valid='test')\n",
|
|
"learn = text_classifier_learner(dls, AWD_LSTM, drop_mult=0.5, metrics=accuracy)\n",
|
|
"projector_word_embeddings(learn=learn, limit=2000, start=2000)\n",
|
|
"```"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "55eefd13",
|
|
"metadata": {},
|
|
"source": [
|
|
"For other language models - like the ones in the **transformers library** - you'll have to pass the layer and vocab. Here's an example for a **BERT** model.\n",
|
|
"```\n",
|
|
"from transformers import AutoTokenizer, AutoModel\n",
|
|
"tokenizer = AutoTokenizer.from_pretrained(\"bert-base-uncased\")\n",
|
|
"model = AutoModel.from_pretrained(\"bert-base-uncased\")\n",
|
|
"\n",
|
|
"# get the word embedding layer\n",
|
|
"layer = model.embeddings.word_embeddings\n",
|
|
"\n",
|
|
"# get and sort vocab\n",
|
|
"vocab_dict = tokenizer.get_vocab()\n",
|
|
"vocab = [k for k, v in sorted(vocab_dict.items(), key=lambda x: x[1])]\n",
|
|
"\n",
|
|
"# write the embeddings for tb projector\n",
|
|
"projector_word_embeddings(layer=layer, vocab=vocab, limit=2000, start=2000)\n",
|
|
"```"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "54b18d89",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| export\n",
|
|
"import tensorboard\n",
|
|
"from torch.utils.tensorboard import SummaryWriter\n",
|
|
"from fastai.callback.fp16 import ModelToHalf\n",
|
|
"from fastai.callback.hook import hook_output"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "b94e6880",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| export\n",
|
|
"class TensorBoardBaseCallback(Callback):\n",
|
|
" order = Recorder.order+1\n",
|
|
" \"Base class for tensorboard callbacks\"\n",
|
|
" def __init__(self): self.run_projector = False\n",
|
|
" \n",
|
|
" def after_pred(self):\n",
|
|
" if self.run_projector: self.feat = _add_projector_features(self.learn, self.h, self.feat)\n",
|
|
" \n",
|
|
" def after_validate(self):\n",
|
|
" if not self.run_projector: return\n",
|
|
" self.run_projector = False\n",
|
|
" self._remove()\n",
|
|
" _write_projector_embedding(self.learn, self.writer, self.feat)\n",
|
|
" \n",
|
|
" def after_fit(self): \n",
|
|
" if self.run: self.writer.close()\n",
|
|
" \n",
|
|
" def _setup_projector(self):\n",
|
|
" self.run_projector = True\n",
|
|
" self.h = hook_output(self.learn.model[1][1] if not self.layer else self.layer)\n",
|
|
" self.feat = {}\n",
|
|
" \n",
|
|
" def _setup_writer(self): self.writer = SummaryWriter(log_dir=self.log_dir)\n",
|
|
" def __del__(self): self._remove()\n",
|
|
" def _remove(self):\n",
|
|
" if getattr(self, 'h', None): self.h.remove()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "f5bbe3c2",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/markdown": [
|
|
"---\n",
|
|
"\n",
|
|
"### TensorBoardBaseCallback\n",
|
|
"\n",
|
|
"> TensorBoardBaseCallback ()\n",
|
|
"\n",
|
|
"Basic class handling tweaks of the training loop by changing a `Learner` in various events"
|
|
],
|
|
"text/plain": [
|
|
"<nbdev.showdoc.BasicMarkdownRenderer>"
|
|
]
|
|
},
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"show_doc(TensorBoardBaseCallback)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "261d75bf",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| export\n",
|
|
"class TensorBoardCallback(TensorBoardBaseCallback):\n",
|
|
" \"Saves model topology, losses & metrics for tensorboard and tensorboard projector during training\"\n",
|
|
" def __init__(self, log_dir=None, trace_model=True, log_preds=True, n_preds=9, projector=False, layer=None):\n",
|
|
" super().__init__()\n",
|
|
" store_attr()\n",
|
|
"\n",
|
|
" def before_fit(self):\n",
|
|
" self.run = not hasattr(self.learn, 'lr_finder') and not hasattr(self, \"gather_preds\") and rank_distrib()==0\n",
|
|
" if not self.run: return\n",
|
|
" self._setup_writer()\n",
|
|
" if self.trace_model:\n",
|
|
" if hasattr(self.learn, 'mixed_precision'):\n",
|
|
" raise Exception(\"Can't trace model in mixed precision, pass `trace_model=False` or don't use FP16.\")\n",
|
|
" b = self.dls.one_batch()\n",
|
|
" self.learn._split(b)\n",
|
|
" self.writer.add_graph(self.model, *self.xb)\n",
|
|
"\n",
|
|
" def after_batch(self):\n",
|
|
" self.writer.add_scalar('train_loss', self.smooth_loss, self.train_iter)\n",
|
|
" for i,h in enumerate(self.opt.hypers):\n",
|
|
" for k,v in h.items(): self.writer.add_scalar(f'{k}_{i}', v, self.train_iter)\n",
|
|
"\n",
|
|
" def after_epoch(self):\n",
|
|
" for n,v in zip(self.recorder.metric_names[2:-1], self.recorder.log[2:-1]):\n",
|
|
" self.writer.add_scalar(n, v, self.train_iter)\n",
|
|
" if self.log_preds:\n",
|
|
" b = self.dls.valid.one_batch()\n",
|
|
" self.learn.one_batch(0, b)\n",
|
|
" preds = getcallable(self.loss_func, 'activation')(self.pred)\n",
|
|
" out = getcallable(self.loss_func, 'decodes')(preds)\n",
|
|
" x,y,its,outs = self.dls.valid.show_results(b, out, show=False, max_n=self.n_preds)\n",
|
|
" tensorboard_log(x, y, its, outs, self.writer, self.train_iter)\n",
|
|
" \n",
|
|
" def before_validate(self):\n",
|
|
" if self.projector: self._setup_projector()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "f9bcf432",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/markdown": [
|
|
"---\n",
|
|
"\n",
|
|
"### TensorBoardCallback\n",
|
|
"\n",
|
|
"> TensorBoardCallback (log_dir=None, trace_model=True, log_preds=True,\n",
|
|
"> n_preds=9, projector=False, layer=None)\n",
|
|
"\n",
|
|
"Saves model topology, losses & metrics for tensorboard and tensorboard projector during training"
|
|
],
|
|
"text/plain": [
|
|
"<nbdev.showdoc.BasicMarkdownRenderer>"
|
|
]
|
|
},
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"show_doc(TensorBoardCallback)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "8b9c90fc",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| export\n",
|
|
"class TensorBoardProjectorCallback(TensorBoardBaseCallback):\n",
|
|
" \"Extracts and exports image featuers for tensorboard projector during inference\"\n",
|
|
" def __init__(self, log_dir=None, layer=None):\n",
|
|
" super().__init__()\n",
|
|
" store_attr()\n",
|
|
" \n",
|
|
" def before_fit(self):\n",
|
|
" self.run = not hasattr(self.learn, 'lr_finder') and hasattr(self, \"gather_preds\") and rank_distrib()==0\n",
|
|
" if not self.run: return\n",
|
|
" self._setup_writer()\n",
|
|
"\n",
|
|
" def before_validate(self):\n",
|
|
" self._setup_projector()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "d6000307",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/markdown": [
|
|
"---\n",
|
|
"\n",
|
|
"### TensorBoardProjectorCallback\n",
|
|
"\n",
|
|
"> TensorBoardProjectorCallback (log_dir=None, layer=None)\n",
|
|
"\n",
|
|
"Extracts and exports image featuers for tensorboard projector during inference"
|
|
],
|
|
"text/plain": [
|
|
"<nbdev.showdoc.BasicMarkdownRenderer>"
|
|
]
|
|
},
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"show_doc(TensorBoardProjectorCallback)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "a8757c5a",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| export\n",
|
|
"def _write_projector_embedding(learn, writer, feat):\n",
|
|
" lbls = [learn.dl.vocab[l] for l in feat['lbl']] if getattr(learn.dl, 'vocab', None) else None \n",
|
|
" vecs = feat['vec'].squeeze()\n",
|
|
" writer.add_embedding(vecs, metadata=lbls, label_img=feat['img'], global_step=learn.train_iter)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "fd753d32",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| export\n",
|
|
"def _add_projector_features(learn, hook, feat):\n",
|
|
" img = _normalize_for_projector(learn.x)\n",
|
|
" first_epoch = True if learn.iter == 0 else False\n",
|
|
" feat['vec'] = hook.stored if first_epoch else torch.cat((feat['vec'], hook.stored),0)\n",
|
|
" feat['img'] = img if first_epoch else torch.cat((feat['img'], img),0)\n",
|
|
" if getattr(learn.dl, 'vocab', None):\n",
|
|
" feat['lbl'] = learn.y if first_epoch else torch.cat((feat['lbl'], learn.y),0)\n",
|
|
" return feat"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "787632a0",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| export\n",
|
|
"def _get_embeddings(model, layer):\n",
|
|
" layer = model[0].encoder if layer == None else layer\n",
|
|
" return layer.weight"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "965c546d",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| export\n",
|
|
"@dispatch\n",
|
|
"def _normalize_for_projector(x:TensorImage):\n",
|
|
" # normalize tensor to be between 0-1\n",
|
|
" img = x.clone()\n",
|
|
" sz = img.shape\n",
|
|
" img = img.view(x.size(0), -1)\n",
|
|
" img -= img.min(1, keepdim=True)[0]\n",
|
|
" img /= img.max(1, keepdim=True)[0]\n",
|
|
" img = img.view(*sz)\n",
|
|
" return img"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "1f0a3ae1",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| export\n",
|
|
"from fastai.text.all import LMLearner, TextLearner"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "80d48601",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| export\n",
|
|
"def projector_word_embeddings(learn=None, layer=None, vocab=None, limit=-1, start=0, log_dir=None):\n",
|
|
" \"Extracts and exports word embeddings from language models embedding layers\"\n",
|
|
" if not layer:\n",
|
|
" if isinstance(learn, LMLearner): layer = learn.model[0].encoder\n",
|
|
" elif isinstance(learn, TextLearner): layer = learn.model[0].module.encoder\n",
|
|
" emb = layer.weight\n",
|
|
" img = torch.full((len(emb),3,8,8), 0.7)\n",
|
|
" vocab = learn.dls.vocab[0] if vocab == None else vocab\n",
|
|
" vocab = list(map(lambda x: f'{x}_', vocab))\n",
|
|
" writer = SummaryWriter(log_dir=log_dir)\n",
|
|
" end = start + limit if limit >= 0 else -1\n",
|
|
" writer.add_embedding(emb[start:end], metadata=vocab[start:end], label_img=img[start:end])\n",
|
|
" writer.close()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "9ee5ffde",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/markdown": [
|
|
"---\n",
|
|
"\n",
|
|
"#### projector_word_embeddings\n",
|
|
"\n",
|
|
"> projector_word_embeddings (learn=None, layer=None, vocab=None, limit=-1,\n",
|
|
"> start=0, log_dir=None)\n",
|
|
"\n",
|
|
"Extracts and exports word embeddings from language models embedding layers"
|
|
],
|
|
"text/plain": [
|
|
"<nbdev.showdoc.BasicMarkdownRenderer>"
|
|
]
|
|
},
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"show_doc(projector_word_embeddings)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "c7528e24",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| export\n",
|
|
"from fastai.vision.data import *"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "6247bca8",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| export\n",
|
|
"@dispatch\n",
|
|
"def tensorboard_log(x:TensorImage, y: TensorCategory, samples, outs, writer, step):\n",
|
|
" fig,axs = get_grid(len(samples), return_fig=True)\n",
|
|
" for i in range(2):\n",
|
|
" axs = [b.show(ctx=c) for b,c in zip(samples.itemgot(i),axs)]\n",
|
|
" axs = [r.show(ctx=c, color='green' if b==r else 'red')\n",
|
|
" for b,r,c in zip(samples.itemgot(1),outs.itemgot(0),axs)]\n",
|
|
" writer.add_figure('Sample results', fig, step)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "ac46fb3d",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| export\n",
|
|
"from fastai.vision.core import TensorPoint,TensorBBox"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "1073635f",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| export\n",
|
|
"@dispatch\n",
|
|
"def tensorboard_log(x:TensorImage, y: TensorImageBase|TensorPoint|TensorBBox, samples, outs, writer, step):\n",
|
|
" fig,axs = get_grid(len(samples), return_fig=True, double=True)\n",
|
|
" for i in range(2):\n",
|
|
" axs[::2] = [b.show(ctx=c) for b,c in zip(samples.itemgot(i),axs[::2])]\n",
|
|
" for x in [samples,outs]:\n",
|
|
" axs[1::2] = [b.show(ctx=c) for b,c in zip(x.itemgot(0),axs[1::2])]\n",
|
|
" writer.add_figure('Sample results', fig, step)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "81f91c13",
|
|
"metadata": {},
|
|
"source": [
|
|
"## TensorBoardCallback"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "42dbd316",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from fastai.vision.all import Resize, RandomSubsetSplitter, aug_transforms, vision_learner, resnet18"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "aa6c8c54",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"path = untar_data(URLs.PETS)\n",
|
|
"\n",
|
|
"db = DataBlock(blocks=(ImageBlock, CategoryBlock), \n",
|
|
" get_items=get_image_files, \n",
|
|
" item_tfms=Resize(128),\n",
|
|
" splitter=RandomSubsetSplitter(train_sz=0.1, valid_sz=0.01),\n",
|
|
" batch_tfms=aug_transforms(size=64),\n",
|
|
" get_y=using_attr(RegexLabeller(r'(.+)_\\d+.*$'), 'name'))\n",
|
|
"\n",
|
|
"dls = db.dataloaders(path/'images')"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "d4e544e5",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"learn = vision_learner(dls, resnet18, metrics=accuracy)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "b4b48247",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/html": [
|
|
"<table border=\"1\" class=\"dataframe\">\n",
|
|
" <thead>\n",
|
|
" <tr style=\"text-align: left;\">\n",
|
|
" <th>epoch</th>\n",
|
|
" <th>train_loss</th>\n",
|
|
" <th>valid_loss</th>\n",
|
|
" <th>accuracy</th>\n",
|
|
" <th>time</th>\n",
|
|
" </tr>\n",
|
|
" </thead>\n",
|
|
" <tbody>\n",
|
|
" <tr>\n",
|
|
" <td>0</td>\n",
|
|
" <td>4.973294</td>\n",
|
|
" <td>5.009670</td>\n",
|
|
" <td>0.082192</td>\n",
|
|
" <td>00:03</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <td>1</td>\n",
|
|
" <td>4.382769</td>\n",
|
|
" <td>4.438282</td>\n",
|
|
" <td>0.095890</td>\n",
|
|
" <td>00:03</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <td>2</td>\n",
|
|
" <td>3.877172</td>\n",
|
|
" <td>3.665855</td>\n",
|
|
" <td>0.178082</td>\n",
|
|
" <td>00:04</td>\n",
|
|
" </tr>\n",
|
|
" </tbody>\n",
|
|
"</table>"
|
|
],
|
|
"text/plain": [
|
|
"<IPython.core.display.HTML object>"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
}
|
|
],
|
|
"source": [
|
|
"learn.unfreeze()\n",
|
|
"learn.fit_one_cycle(3, cbs=TensorBoardCallback(Path.home()/'tmp'/'runs'/'tb', trace_model=True))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "12f7ec72",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Projector"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "24a096b3",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Projector in TensorBoardCallback"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "51621929",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"path = untar_data(URLs.PETS)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "0ad367ec",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"db = DataBlock(blocks=(ImageBlock, CategoryBlock), \n",
|
|
" get_items=get_image_files, \n",
|
|
" item_tfms=Resize(128),\n",
|
|
" splitter=RandomSubsetSplitter(train_sz=0.05, valid_sz=0.01),\n",
|
|
" batch_tfms=aug_transforms(size=64),\n",
|
|
" get_y=using_attr(RegexLabeller(r'(.+)_\\d+.*$'), 'name'))\n",
|
|
"\n",
|
|
"dls = db.dataloaders(path/'images')"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "cbabf19e",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"cbs = [TensorBoardCallback(log_dir=Path.home()/'tmp'/'runs'/'vision1', projector=True)]\n",
|
|
"learn = vision_learner(dls, resnet18, metrics=accuracy)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "312befba",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/html": [
|
|
"<table border=\"1\" class=\"dataframe\">\n",
|
|
" <thead>\n",
|
|
" <tr style=\"text-align: left;\">\n",
|
|
" <th>epoch</th>\n",
|
|
" <th>train_loss</th>\n",
|
|
" <th>valid_loss</th>\n",
|
|
" <th>accuracy</th>\n",
|
|
" <th>time</th>\n",
|
|
" </tr>\n",
|
|
" </thead>\n",
|
|
" <tbody>\n",
|
|
" <tr>\n",
|
|
" <td>0</td>\n",
|
|
" <td>5.143322</td>\n",
|
|
" <td>6.736727</td>\n",
|
|
" <td>0.082192</td>\n",
|
|
" <td>00:03</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <td>1</td>\n",
|
|
" <td>4.508100</td>\n",
|
|
" <td>5.106580</td>\n",
|
|
" <td>0.109589</td>\n",
|
|
" <td>00:03</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <td>2</td>\n",
|
|
" <td>4.057889</td>\n",
|
|
" <td>4.194602</td>\n",
|
|
" <td>0.068493</td>\n",
|
|
" <td>00:03</td>\n",
|
|
" </tr>\n",
|
|
" </tbody>\n",
|
|
"</table>"
|
|
],
|
|
"text/plain": [
|
|
"<IPython.core.display.HTML object>"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
}
|
|
],
|
|
"source": [
|
|
"learn.unfreeze()\n",
|
|
"learn.fit_one_cycle(3, cbs=cbs)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "d3c2b436",
|
|
"metadata": {},
|
|
"source": [
|
|
"### TensorBoardProjectorCallback"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "4b749651",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"path = untar_data(URLs.PETS)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "a679b8b3",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"db = DataBlock(blocks=(ImageBlock, CategoryBlock), \n",
|
|
" get_items=get_image_files, \n",
|
|
" item_tfms=Resize(128),\n",
|
|
" splitter=RandomSubsetSplitter(train_sz=0.1, valid_sz=0.01),\n",
|
|
" batch_tfms=aug_transforms(size=64),\n",
|
|
" get_y=using_attr(RegexLabeller(r'(.+)_\\d+.*$'), 'name'))\n",
|
|
"\n",
|
|
"dls = db.dataloaders(path/'images')"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "ec666adf",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"files = get_image_files(path/'images')\n",
|
|
"files = files[:256]"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "b125694d",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"dl = learn.dls.test_dl(files, with_labels=True)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "c48fb283",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"learn = vision_learner(dls, resnet18, metrics=accuracy)\n",
|
|
"layer = learn.model[1][0].ap\n",
|
|
"cbs = [TensorBoardProjectorCallback(layer=layer, log_dir=Path.home()/'tmp'/'runs'/'vision2')]"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "0f2136f7",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/html": [],
|
|
"text/plain": [
|
|
"<IPython.core.display.HTML object>"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
}
|
|
],
|
|
"source": [
|
|
"_ = learn.get_preds(dl=dl, cbs=cbs)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "2c8ff1eb",
|
|
"metadata": {},
|
|
"source": [
|
|
"## projector_word_embeddings"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "2520d937",
|
|
"metadata": {},
|
|
"source": [
|
|
"### fastai text or lm learner"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "d28ab6fb",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from fastai.text.all import TextDataLoaders, text_classifier_learner, AWD_LSTM"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "041e1521",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"dls = TextDataLoaders.from_folder(untar_data(URLs.IMDB), valid='test')\n",
|
|
"learn = text_classifier_learner(dls, AWD_LSTM, drop_mult=0.5, metrics=accuracy)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "67e8e891",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"projector_word_embeddings(learn, limit=1000, log_dir=Path.home()/'tmp'/'runs'/'text')"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "efd79ded",
|
|
"metadata": {},
|
|
"source": [
|
|
"### transformers"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "6dcfab5b",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### GPT2"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "5d7cac4e",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from transformers import GPT2LMHeadModel, GPT2TokenizerFast"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "4277d4e1",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"tokenizer = GPT2TokenizerFast.from_pretrained('gpt2')\n",
|
|
"model = GPT2LMHeadModel.from_pretrained('gpt2')\n",
|
|
"layer = model.transformer.wte\n",
|
|
"vocab_dict = tokenizer.get_vocab()\n",
|
|
"vocab = [k for k, v in sorted(vocab_dict.items(), key=lambda x: x[1])]\n",
|
|
"\n",
|
|
"projector_word_embeddings(layer=layer, vocab=vocab, limit=2000, log_dir=Path.home()/'tmp'/'runs'/'transformers')"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "5ac5735f",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### BERT"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "bec8a8dd",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from transformers import AutoTokenizer, AutoModel"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "b876c7fc",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"warning: Embedding dir exists, did you set global_step for add_embedding()?\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"tokenizer = AutoTokenizer.from_pretrained(\"bert-base-uncased\")\n",
|
|
"model = AutoModel.from_pretrained(\"bert-base-uncased\")\n",
|
|
"\n",
|
|
"layer = model.embeddings.word_embeddings\n",
|
|
"\n",
|
|
"vocab_dict = tokenizer.get_vocab()\n",
|
|
"vocab = [k for k, v in sorted(vocab_dict.items(), key=lambda x: x[1])]\n",
|
|
"\n",
|
|
"projector_word_embeddings(layer=layer, vocab=vocab, limit=2000, start=2000, log_dir=Path.home()/'tmp'/'runs'/'transformers')"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "3b71441a",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Validate results in tensorboard"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ea3091ba",
|
|
"metadata": {},
|
|
"source": [
|
|
"Run the following command in the command line to check if the projector embeddings have been correctly wirtten:\n",
|
|
"\n",
|
|
"```\n",
|
|
"tensorboard --logdir=~/tmp/runs\n",
|
|
"```\n",
|
|
"\n",
|
|
"Open http://localhost:6006 in browser (TensorBoard Projector doesn't work correctly in Safari!)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "d80a0d05",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Export -"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "6d587ad9",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| hide\n",
|
|
"from nbdev import *\n",
|
|
"nbdev_export()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "a9a1783d",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": []
|
|
}
|
|
],
|
|
"metadata": {
|
|
"jupytext": {
|
|
"split_at_heading": true
|
|
},
|
|
"kernelspec": {
|
|
"display_name": "python3",
|
|
"language": "python",
|
|
"name": "python3"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|