wehub-resource-sync
1083 lines
38 KiB
Plaintext
1083 lines
38 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "c6f5cf86",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| hide\n",
|
|
"#| eval: false\n",
|
|
"! [ -e /content ] && pip install -Uqq fastai # upgrade fastai on colab"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "51b6b59a",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| default_exp callback.core"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "93fea1ce",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| export\n",
|
|
"from fastai.data.all import *\n",
|
|
"from fastai.optimizer import *\n",
|
|
"from fastai.losses import BaseLoss"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "e42348ff",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| hide\n",
|
|
"from nbdev.showdoc import *"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "80cdee5a",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| export\n",
|
|
"_all_ = ['CancelStepException','CancelBackwardException','CancelFitException','CancelEpochException','CancelTrainException','CancelValidException','CancelBatchException']"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "28d52721",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Callbacks\n",
|
|
"\n",
|
|
"> Basic callbacks for Learner"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "616c8998",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Events"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "37d75a45",
|
|
"metadata": {},
|
|
"source": [
|
|
"Callbacks can occur at any of these times:: *after_create before_fit before_epoch before_train before_batch after_pred after_loss before_backward after_cancel_backward after_backward before_step after_cancel_step after_step after_cancel_batch after_batch after_cancel_train after_train before_validate after_cancel_validate after_validate after_cancel_epoch after_epoch after_cancel_fit after_fit*."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "ab22dd60",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| export\n",
|
|
"_events = L.split('after_create before_fit before_epoch before_train before_batch after_pred after_loss \\\n",
|
|
" before_backward after_cancel_backward after_backward before_step after_cancel_step after_step \\\n",
|
|
" after_cancel_batch after_batch after_cancel_train after_train before_validate after_cancel_validate \\\n",
|
|
" after_validate after_cancel_epoch after_epoch after_cancel_fit after_fit')\n",
|
|
"\n",
|
|
"mk_class('event', **_events.map_dict(),\n",
|
|
" doc=\"All possible events as attributes to get tab-completion and typo-proofing\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "0ed2aba5",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| export\n",
|
|
"_all_ = ['event']"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "739d2d43",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/markdown": [
|
|
"<h3 id=\"event\" class=\"doc_header\"><code>class</code> <code>event</code><a href=\"\" class=\"source_link\" style=\"float:right\">[source]</a></h3>\n",
|
|
"\n",
|
|
"> <code>event</code>(**\\*`args`**, **\\*\\*`kwargs`**)\n",
|
|
"\n",
|
|
"All possible events as attributes to get tab-completion and typo-proofing"
|
|
],
|
|
"text/plain": [
|
|
"<IPython.core.display.Markdown object>"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
}
|
|
],
|
|
"source": [
|
|
"show_doc(event, name='event', title_level=3)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "06bac1db",
|
|
"metadata": {},
|
|
"source": [
|
|
"To ensure that you are referring to an event (that is, the name of one of the times when callbacks are called) that exists, and to get tab completion of event names, use `event`:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "94eefa58",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"test_eq(event.before_step, 'before_step')"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "64a50601",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Callback - "
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "05cd2800",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| export\n",
|
|
"_inner_loop = \"before_batch after_pred after_loss before_backward after_cancel_backward after_backward before_step after_step after_cancel_batch after_batch\".split()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "9eba90b5",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| export\n",
|
|
"_ex_docs = dict(\n",
|
|
" CancelBatchException=\"Skip the rest of this batch and go to `after_batch`\",\n",
|
|
" CancelTrainException=\"Skip the rest of the training part of the epoch and go to `after_train`\",\n",
|
|
" CancelValidException=\"Skip the rest of the validation part of the epoch and go to `after_validate`\",\n",
|
|
" CancelEpochException=\"Skip the rest of this epoch and go to `after_epoch`\",\n",
|
|
" CancelStepException =\"Skip stepping the optimizer\",\n",
|
|
" CancelBackwardException=\"Skip the backward pass and go to `after_backward`\",\n",
|
|
" CancelFitException =\"Interrupts training and go to `after_fit`\")\n",
|
|
"\n",
|
|
"for c,d in _ex_docs.items(): mk_class(c,sup=Exception,doc=d)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "011373d8",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| export\n",
|
|
"@funcs_kwargs(as_method=True)\n",
|
|
"class Callback(Stateful,GetAttr):\n",
|
|
" \"Basic class handling tweaks of the training loop by changing a `Learner` in various events\"\n",
|
|
" order,_default,learn,run,run_train,run_valid = 0,'learn',None,True,True,True\n",
|
|
" _methods = _events\n",
|
|
"\n",
|
|
" def __init__(self, **kwargs): assert not kwargs, f'Passed unknown events: {kwargs}'\n",
|
|
" def __repr__(self): return type(self).__name__\n",
|
|
"\n",
|
|
" def __call__(self, event_name):\n",
|
|
" \"Call `self.{event_name}` if it's defined\"\n",
|
|
" _run = (event_name not in _inner_loop or (self.run_train and getattr(self, 'training', True)) or\n",
|
|
" (self.run_valid and not getattr(self, 'training', False)))\n",
|
|
" res = None\n",
|
|
" if self.run and _run: \n",
|
|
" try: res = getcallable(self, event_name)()\n",
|
|
" except (CancelBatchException, CancelBackwardException, CancelEpochException, CancelFitException, CancelStepException, CancelTrainException, CancelValidException): raise\n",
|
|
" except Exception as e: raise modify_exception(e, f'Exception occured in `{self.__class__.__name__}` when calling event `{event_name}`:\\n\\t{e.args[0]}', replace=True)\n",
|
|
" if event_name=='after_fit': self.run=True #Reset self.run to True at each end of fit\n",
|
|
" return res\n",
|
|
"\n",
|
|
" def __setattr__(self, name, value):\n",
|
|
" \"Set an attribute for a `Callback`\"\n",
|
|
" if hasattr(self.learn,name):\n",
|
|
" warn(f\"You are shadowing an attribute ({name}) that exists in the learner. Use `self.learn.{name}` to avoid this\")\n",
|
|
" super().__setattr__(name, value)\n",
|
|
"\n",
|
|
" @property\n",
|
|
" def name(self):\n",
|
|
" \"Name of the `Callback`, camel-cased and with '*Callback*' removed\"\n",
|
|
" return class2attr(self, 'Callback')"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "f24e2b71",
|
|
"metadata": {},
|
|
"source": [
|
|
"The training loop is defined in `Learner` a bit below and consists in a minimal set of instructions: looping through the data we:\n",
|
|
"\n",
|
|
"- compute the output of the model from the input\n",
|
|
"- calculate a loss between this output and the desired target\n",
|
|
"- compute the gradients of this loss with respect to all the model parameters\n",
|
|
"- update the parameters accordingly\n",
|
|
"- zero all the gradients\n",
|
|
"\n",
|
|
"Any tweak of this training loop is defined in a `Callback` to avoid over-complicating the code of the training loop, and to make it easy to mix and match different techniques (since they'll be defined in different callbacks). A callback can implement actions on the following events:\n",
|
|
"\n",
|
|
"- `after_create`: called after the `Learner` is created\n",
|
|
"- `before_fit`: called before starting training or inference, ideal for initial setup.\n",
|
|
"- `before_epoch`: called at the beginning of each epoch, useful for any behavior you need to reset at each epoch.\n",
|
|
"- `before_train`: called at the beginning of the training part of an epoch.\n",
|
|
"- `before_batch`: called at the beginning of each batch, just after drawing said batch. It can be used to do any setup necessary for the batch (like hyper-parameter scheduling) or to change the input/target before it goes in the model (change of the input with techniques like mixup for instance).\n",
|
|
"- `after_pred`: called after computing the output of the model on the batch. It can be used to change that output before it's fed to the loss.\n",
|
|
"- `after_loss`: called after the loss has been computed, but before the backward pass. It can be used to add any penalty to the loss (AR or TAR in RNN training for instance).\n",
|
|
"- `before_backward`: called after the loss has been computed, but only in training mode (i.e. when the backward pass will be used)\n",
|
|
"- `after_backward`: called after the backward pass, but before the update of the parameters. Generally `before_step` should be used instead.\n",
|
|
"- `before_step`: called after the backward pass, but before the update of the parameters. It can be used to do any change to the gradients before said update (gradient clipping for instance).\n",
|
|
"- `after_step`: called after the step and before the gradients are zeroed.\n",
|
|
"- `after_batch`: called at the end of a batch, for any clean-up before the next one.\n",
|
|
"- `after_train`: called at the end of the training phase of an epoch.\n",
|
|
"- `before_validate`: called at the beginning of the validation phase of an epoch, useful for any setup needed specifically for validation.\n",
|
|
"- `after_validate`: called at the end of the validation part of an epoch.\n",
|
|
"- `after_epoch`: called at the end of an epoch, for any clean-up before the next one.\n",
|
|
"- `after_fit`: called at the end of training, for final clean-up."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "72e3ce45",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/markdown": [
|
|
"<h4 id=\"Callback.__call__\" class=\"doc_header\"><code>Callback.__call__</code><a href=\"__main__.py#L11\" class=\"source_link\" style=\"float:right\">[source]</a></h4>\n",
|
|
"\n",
|
|
"> <code>Callback.__call__</code>(**`event_name`**)\n",
|
|
"\n",
|
|
"Call `self.{event_name}` if it's defined"
|
|
],
|
|
"text/plain": [
|
|
"<IPython.core.display.Markdown object>"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
}
|
|
],
|
|
"source": [
|
|
"show_doc(Callback.__call__)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "7b95b65b",
|
|
"metadata": {},
|
|
"source": [
|
|
"One way to define callbacks is through subclassing:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "9446ae18",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"class _T(Callback):\n",
|
|
" def call_me(self): return \"maybe\"\n",
|
|
"test_eq(_T()(\"call_me\"), \"maybe\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "196fdf0f",
|
|
"metadata": {},
|
|
"source": [
|
|
"Another way is by passing the callback function to the constructor:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "18ae6642",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"def cb(self): return \"maybe\"\n",
|
|
"_t = Callback(before_fit=cb)\n",
|
|
"test_eq(_t(event.before_fit), \"maybe\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "fb79f81d",
|
|
"metadata": {},
|
|
"source": [
|
|
"`Callback`s provide a shortcut to avoid having to write `self.learn.bla` for any `bla` attribute we seek; instead, just write `self.bla`. This only works for getting attributes, *not* for setting them."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "8141f616",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"mk_class('TstLearner', 'a')\n",
|
|
"\n",
|
|
"class TstCallback(Callback):\n",
|
|
" def batch_begin(self): print(self.a)\n",
|
|
"\n",
|
|
"learn,cb = TstLearner(1),TstCallback()\n",
|
|
"cb.learn = learn\n",
|
|
"test_stdout(lambda: cb('batch_begin'), \"1\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "3ace1bcb",
|
|
"metadata": {},
|
|
"source": [
|
|
"If you want to change the value of an attribute, you have to use `self.learn.bla`, no `self.bla`. In the example below, `self.a += 1` creates an `a` attribute of 2 in the callback instead of setting the `a` of the learner to 2. It also issues a warning that something is probably wrong:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "74a1e465",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"1"
|
|
]
|
|
},
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"learn.a"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "25125d00",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"/tmp/ipykernel_5201/1369389649.py:29: UserWarning: You are shadowing an attribute (a) that exists in the learner. Use `self.learn.a` to avoid this\n",
|
|
" warn(f\"You are shadowing an attribute ({name}) that exists in the learner. Use `self.learn.{name}` to avoid this\")\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"class TstCallback(Callback):\n",
|
|
" def batch_begin(self): self.a += 1\n",
|
|
"\n",
|
|
"learn,cb = TstLearner(1),TstCallback()\n",
|
|
"cb.learn = learn\n",
|
|
"cb('batch_begin')\n",
|
|
"test_eq(cb.a, 2)\n",
|
|
"test_eq(cb.learn.a, 1)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "88595384",
|
|
"metadata": {},
|
|
"source": [
|
|
"A proper version needs to write `self.learn.a = self.a + 1`:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "1644028d",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"class TstCallback(Callback):\n",
|
|
" def batch_begin(self): self.learn.a = self.a + 1\n",
|
|
"\n",
|
|
"learn,cb = TstLearner(1),TstCallback()\n",
|
|
"cb.learn = learn\n",
|
|
"cb('batch_begin')\n",
|
|
"test_eq(cb.learn.a, 2)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "36fb2de6",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| hide\n",
|
|
"class TstCallback(Callback):\n",
|
|
" def batch_begin(self): self.learn.a = 1 + \"a\"\n",
|
|
"learn,cb = TstLearner(1),TstCallback()\n",
|
|
"cb.learn = learn\n",
|
|
"with ExceptionExpected(TypeError, regex=\" in `TstCallback` when calling event `batch_begin`\"):\n",
|
|
" cb('batch_begin')"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "e9801458",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/markdown": [
|
|
"<h4 id=\"Callback.name\" class=\"doc_header\"><code>Callback.name</code><a href=\"\" class=\"source_link\" style=\"float:right\">[source]</a></h4>\n",
|
|
"\n",
|
|
"Name of the [`Callback`](/callback.core.html#Callback), camel-cased and with '*Callback*' removed"
|
|
],
|
|
"text/plain": [
|
|
"<IPython.core.display.Markdown object>"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
}
|
|
],
|
|
"source": [
|
|
"show_doc(Callback.name, name='Callback.name')"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "eb66657e",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"test_eq(TstCallback().name, 'tst')\n",
|
|
"class ComplicatedNameCallback(Callback): pass\n",
|
|
"test_eq(ComplicatedNameCallback().name, 'complicated_name')"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "3ad1ce0a",
|
|
"metadata": {},
|
|
"source": [
|
|
"## TrainEvalCallback -"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "d8968bed",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| export\n",
|
|
"class TrainEvalCallback(Callback):\n",
|
|
" \"`Callback` that tracks the number of iterations done and properly sets training/eval mode\"\n",
|
|
" order,run_valid = -10,False\n",
|
|
" def after_create(self): self.learn.n_epoch = 1\n",
|
|
"\n",
|
|
" def before_fit(self):\n",
|
|
" \"Set the iter and epoch counters to 0, put the model and the right device\"\n",
|
|
" self.learn.epoch,self.learn.loss = 0,tensor(0.)\n",
|
|
" self.learn.train_iter,self.learn.pct_train = 0,0.\n",
|
|
" device = getattr(self.dls, 'device', default_device())\n",
|
|
" self.model.to(device)\n",
|
|
" if isinstance(self.loss_func, (nn.Module, BaseLoss)): self.loss_func.to(device)\n",
|
|
" if hasattr(self.model, 'reset'): self.model.reset()\n",
|
|
"\n",
|
|
" def after_batch(self):\n",
|
|
" \"Update the iter counter (in training mode)\"\n",
|
|
" self.learn.pct_train += 1./(self.n_iter*self.n_epoch)\n",
|
|
" self.learn.train_iter += 1\n",
|
|
"\n",
|
|
" def before_train(self):\n",
|
|
" \"Set the model to training mode\"\n",
|
|
" self.learn.pct_train=self.epoch/self.n_epoch\n",
|
|
" self.model.train()\n",
|
|
" self.learn.training=True\n",
|
|
"\n",
|
|
" def before_validate(self):\n",
|
|
" \"Set the model to validation mode\"\n",
|
|
" self.model.eval()\n",
|
|
" self.learn.training=False"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "aa38e424",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/markdown": [
|
|
"<h3 id=\"TrainEvalCallback\" class=\"doc_header\"><code>class</code> <code>TrainEvalCallback</code><a href=\"\" class=\"source_link\" style=\"float:right\">[source]</a></h3>\n",
|
|
"\n",
|
|
"> <code>TrainEvalCallback</code>(**`after_create`**=*`None`*, **`before_fit`**=*`None`*, **`before_epoch`**=*`None`*, **`before_train`**=*`None`*, **`before_batch`**=*`None`*, **`after_pred`**=*`None`*, **`after_loss`**=*`None`*, **`before_backward`**=*`None`*, **`before_step`**=*`None`*, **`after_cancel_step`**=*`None`*, **`after_step`**=*`None`*, **`after_cancel_batch`**=*`None`*, **`after_batch`**=*`None`*, **`after_cancel_train`**=*`None`*, **`after_train`**=*`None`*, **`before_validate`**=*`None`*, **`after_cancel_validate`**=*`None`*, **`after_validate`**=*`None`*, **`after_cancel_epoch`**=*`None`*, **`after_epoch`**=*`None`*, **`after_cancel_fit`**=*`None`*, **`after_fit`**=*`None`*) :: [`Callback`](/callback.core.html#Callback)\n",
|
|
"\n",
|
|
"[`Callback`](/callback.core.html#Callback) that tracks the number of iterations done and properly sets training/eval mode"
|
|
],
|
|
"text/plain": [
|
|
"<IPython.core.display.Markdown object>"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
}
|
|
],
|
|
"source": [
|
|
"show_doc(TrainEvalCallback, title_level=3)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "39fba65f",
|
|
"metadata": {},
|
|
"source": [
|
|
"This `Callback` is automatically added in every `Learner` at initialization."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "f59d301d",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| hide\n",
|
|
"#test of the TrainEvalCallback below in Learner.fit"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "33328a26",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| export\n",
|
|
"if not hasattr(defaults, 'callbacks'): defaults.callbacks = [TrainEvalCallback]"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "86b7bf91",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Attributes available to callbacks"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "d56e01c6",
|
|
"metadata": {},
|
|
"source": [
|
|
"When writing a callback, the following attributes of `Learner` are available:\n",
|
|
"\n",
|
|
"- `model`: the model used for training/validation\n",
|
|
"- `dls`: the underlying `DataLoaders`\n",
|
|
"- `loss_func`: the loss function used\n",
|
|
"- `opt`: the optimizer used to update the model parameters\n",
|
|
"- `opt_func`: the function used to create the optimizer\n",
|
|
"- `cbs`: the list containing all `Callback`s\n",
|
|
"- `dl`: current `DataLoader` used for iteration\n",
|
|
"- `x`/`xb`: last input drawn from `self.dl` (potentially modified by callbacks). `xb` is always a tuple (potentially with one element) and `x` is detuplified. You can only assign to `xb`.\n",
|
|
"- `y`/`yb`: last target drawn from `self.dl` (potentially modified by callbacks). `yb` is always a tuple (potentially with one element) and `y` is detuplified. You can only assign to `yb`.\n",
|
|
"- `pred`: last predictions from `self.model` (potentially modified by callbacks)\n",
|
|
"- `loss_grad`: last computed loss (potentially modified by callbacks)\n",
|
|
"- `loss`: clone of `loss_grad` used for logging\n",
|
|
"- `n_epoch`: the number of epochs in this training\n",
|
|
"- `n_iter`: the number of iterations in the current `self.dl`\n",
|
|
"- `epoch`: the current epoch index (from 0 to `n_epoch-1`)\n",
|
|
"- `iter`: the current iteration index in `self.dl` (from 0 to `n_iter-1`)\n",
|
|
"\n",
|
|
"The following attributes are added by `TrainEvalCallback` and should be available unless you went out of your way to remove that callback:\n",
|
|
"\n",
|
|
"- `train_iter`: the number of training iterations done since the beginning of this training\n",
|
|
"- `pct_train`: from 0. to 1., the percentage of training iterations completed\n",
|
|
"- `training`: flag to indicate if we're in training mode or not\n",
|
|
"\n",
|
|
"The following attribute is added by `Recorder` and should be available unless you went out of your way to remove that callback:\n",
|
|
"\n",
|
|
"- `smooth_loss`: an exponentially-averaged version of the training loss"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "b974de7b",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Callbacks control flow"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "901f36f9",
|
|
"metadata": {},
|
|
"source": [
|
|
"It happens that we may want to skip some of the steps of the training loop: in gradient accumulation, we don't always want to do the step/zeroing of the grads for instance. During an LR finder test, we don't want to do the validation phase of an epoch. Or if we're training with a strategy of early stopping, we want to be able to completely interrupt the training loop.\n",
|
|
"\n",
|
|
"This is made possible by raising specific exceptions the training loop will look for (and properly catch)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "16d49b62",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/markdown": [
|
|
"<h3 id=\"CancelStepException\" class=\"doc_header\"><code>class</code> <code>CancelStepException</code><a href=\"\" class=\"source_link\" style=\"float:right\">[source]</a></h3>\n",
|
|
"\n",
|
|
"> <code>CancelStepException</code>(**\\*`args`**, **\\*\\*`kwargs`**) :: `Exception`\n",
|
|
"\n",
|
|
"Skip stepping the optimizer"
|
|
],
|
|
"text/plain": [
|
|
"<IPython.core.display.Markdown object>"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
}
|
|
],
|
|
"source": [
|
|
"show_doc(CancelStepException, title_level=3)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "07e278ca",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/markdown": [
|
|
"<h3 id=\"CancelBatchException\" class=\"doc_header\"><code>class</code> <code>CancelBatchException</code><a href=\"\" class=\"source_link\" style=\"float:right\">[source]</a></h3>\n",
|
|
"\n",
|
|
"> <code>CancelBatchException</code>(**\\*`args`**, **\\*\\*`kwargs`**) :: `Exception`\n",
|
|
"\n",
|
|
"Skip the rest of this batch and go to `after_batch`"
|
|
],
|
|
"text/plain": [
|
|
"<IPython.core.display.Markdown object>"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
}
|
|
],
|
|
"source": [
|
|
"show_doc(CancelBatchException, title_level=3)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "ea1ae31f",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/markdown": [
|
|
"<h3 id=\"CancelBackwardException\" class=\"doc_header\"><code>class</code> <code>CancelBackwardException</code><a href=\"\" class=\"source_link\" style=\"float:right\">[source]</a></h3>\n",
|
|
"\n",
|
|
"> <code>CancelBackwardException</code>(**\\*`args`**, **\\*\\*`kwargs`**) :: `Exception`\n",
|
|
"\n",
|
|
"Skip the backward pass and go to `after_backward`"
|
|
],
|
|
"text/plain": [
|
|
"<IPython.core.display.Markdown object>"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
}
|
|
],
|
|
"source": [
|
|
"show_doc(CancelBackwardException, title_level=3)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "240a0989",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/markdown": [
|
|
"<h3 id=\"CancelTrainException\" class=\"doc_header\"><code>class</code> <code>CancelTrainException</code><a href=\"\" class=\"source_link\" style=\"float:right\">[source]</a></h3>\n",
|
|
"\n",
|
|
"> <code>CancelTrainException</code>(**\\*`args`**, **\\*\\*`kwargs`**) :: `Exception`\n",
|
|
"\n",
|
|
"Skip the rest of the training part of the epoch and go to `after_train`"
|
|
],
|
|
"text/plain": [
|
|
"<IPython.core.display.Markdown object>"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
}
|
|
],
|
|
"source": [
|
|
"show_doc(CancelTrainException, title_level=3)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "55553eb0",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/markdown": [
|
|
"<h3 id=\"CancelValidException\" class=\"doc_header\"><code>class</code> <code>CancelValidException</code><a href=\"\" class=\"source_link\" style=\"float:right\">[source]</a></h3>\n",
|
|
"\n",
|
|
"> <code>CancelValidException</code>(**\\*`args`**, **\\*\\*`kwargs`**) :: `Exception`\n",
|
|
"\n",
|
|
"Skip the rest of the validation part of the epoch and go to `after_validate`"
|
|
],
|
|
"text/plain": [
|
|
"<IPython.core.display.Markdown object>"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
}
|
|
],
|
|
"source": [
|
|
"show_doc(CancelValidException, title_level=3)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "9deb56b4",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/markdown": [
|
|
"<h3 id=\"CancelEpochException\" class=\"doc_header\"><code>class</code> <code>CancelEpochException</code><a href=\"\" class=\"source_link\" style=\"float:right\">[source]</a></h3>\n",
|
|
"\n",
|
|
"> <code>CancelEpochException</code>(**\\*`args`**, **\\*\\*`kwargs`**) :: `Exception`\n",
|
|
"\n",
|
|
"Skip the rest of this epoch and go to `after_epoch`"
|
|
],
|
|
"text/plain": [
|
|
"<IPython.core.display.Markdown object>"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
}
|
|
],
|
|
"source": [
|
|
"show_doc(CancelEpochException, title_level=3)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "b7f06229",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/markdown": [
|
|
"<h3 id=\"CancelFitException\" class=\"doc_header\"><code>class</code> <code>CancelFitException</code><a href=\"\" class=\"source_link\" style=\"float:right\">[source]</a></h3>\n",
|
|
"\n",
|
|
"> <code>CancelFitException</code>(**\\*`args`**, **\\*\\*`kwargs`**) :: `Exception`\n",
|
|
"\n",
|
|
"Interrupts training and go to `after_fit`"
|
|
],
|
|
"text/plain": [
|
|
"<IPython.core.display.Markdown object>"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
}
|
|
],
|
|
"source": [
|
|
"show_doc(CancelFitException, title_level=3)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "4a1e417a",
|
|
"metadata": {},
|
|
"source": [
|
|
"You can detect one of those exceptions occurred and add code that executes right after with the following events:\n",
|
|
"\n",
|
|
"- `after_cancel_batch`: reached immediately after a `CancelBatchException` before proceeding to `after_batch`\n",
|
|
"- `after_cancel_train`: reached immediately after a `CancelTrainException` before proceeding to `after_epoch`\n",
|
|
"- `after_cancel_valid`: reached immediately after a `CancelValidException` before proceeding to `after_epoch`\n",
|
|
"- `after_cancel_epoch`: reached immediately after a `CancelEpochException` before proceeding to `after_epoch`\n",
|
|
"- `after_cancel_fit`: reached immediately after a `CancelFitException` before proceeding to `after_fit`"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "606f3f74",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Gather and fetch preds callbacks -"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "a558f93f",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| export\n",
|
|
"class GatherPredsCallback(Callback):\n",
|
|
" \"`Callback` that returns all predictions and targets, optionally `with_input` or `with_loss`\"\n",
|
|
" _stateattrs=('preds','targets','inputs','losses')\n",
|
|
" def __init__(self,\n",
|
|
" with_input:bool=False, # Whether to return inputs\n",
|
|
" with_loss:bool=False, # Whether to return losses\n",
|
|
" save_preds:Path=None, # Path to save predictions\n",
|
|
" save_targs:Path=None, # Path to save targets\n",
|
|
" with_preds:bool=True, # Whether to return predictions\n",
|
|
" with_targs:bool=True, # Whether to return targets\n",
|
|
" concat_dim:int=0, # Dimension to concatenate returned tensors\n",
|
|
" pickle_protocol:int=2 # Pickle protocol used to save predictions and targets\n",
|
|
" ):\n",
|
|
" store_attr()\n",
|
|
"\n",
|
|
" def before_batch(self):\n",
|
|
" \"If `with_input`, detach batch inputs\"\n",
|
|
" if self.with_input: self.inputs.append((self.learn.to_detach(self.xb)))\n",
|
|
"\n",
|
|
" def before_validate(self):\n",
|
|
" \"Initialize containers\"\n",
|
|
" self.preds,self.targets = [],[]\n",
|
|
" if self.with_input: self.inputs = []\n",
|
|
" if self.with_loss: self.losses = []\n",
|
|
"\n",
|
|
" def after_batch(self):\n",
|
|
" \"Save predictions, targets and potentially losses\"\n",
|
|
" if not hasattr(self, 'pred'): return\n",
|
|
" preds,targs = self.learn.to_detach(self.pred),self.learn.to_detach(self.yb)\n",
|
|
" if self.with_preds: self.preds.append(preds)\n",
|
|
" if self.with_targs: self.targets.append(targs)\n",
|
|
" if self.save_preds is not None: \n",
|
|
" torch.save(preds, self.save_preds/str(self.iter), pickle_protocol=self.pickle_protocol)\n",
|
|
" if self.save_targs is not None: \n",
|
|
" torch.save(targs[0], self.save_targs/str(self.iter), pickle_protocol=self.pickle_protocol)\n",
|
|
" if self.with_loss:\n",
|
|
" bs = find_bs(self.yb)\n",
|
|
" loss = self.loss if self.loss.numel() == bs else self.loss.view(bs,-1).mean(1)\n",
|
|
" self.losses.append(self.learn.to_detach(loss))\n",
|
|
"\n",
|
|
" def after_validate(self):\n",
|
|
" \"Concatenate all recorded tensors\"\n",
|
|
" if not hasattr(self, 'preds'): return\n",
|
|
" if self.with_input: self.inputs = detuplify(to_concat(self.inputs, dim=self.concat_dim))\n",
|
|
" if self.with_preds: self.preds = detuplify(to_concat(self.preds, dim=self.concat_dim))\n",
|
|
" if self.with_targs: self.targets = detuplify(to_concat(self.targets, dim=self.concat_dim))\n",
|
|
" if self.with_loss: self.losses = to_concat(self.losses)\n",
|
|
"\n",
|
|
" def all_tensors(self) -> (Tensor, list):\n",
|
|
" \"Returns all recorded tensors in the order [inputs, preds, targets, losses]\"\n",
|
|
" res = [self.preds if self.with_preds else None, self.targets if self.with_targs else None]\n",
|
|
" if self.with_input: res = [self.inputs] + res\n",
|
|
" if self.with_loss: res.append(self.losses)\n",
|
|
" return res"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "b1dc2582",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/markdown": [
|
|
"<h3 id=\"GatherPredsCallback\" class=\"doc_header\"><code>class</code> <code>GatherPredsCallback</code><a href=\"\" class=\"source_link\" style=\"float:right\">[source]</a></h3>\n",
|
|
"\n",
|
|
"> <code>GatherPredsCallback</code>(**`with_input`**:`bool`=*`False`*, **`with_loss`**:`bool`=*`False`*, **`save_preds`**:`PathLike'>)`=*`None`*, **`save_targs`**:`PathLike'>)`=*`None`*, **`with_preds`**:`bool`=*`True`*, **`with_targs`**:`bool`=*`True`*, **`concat_dim`**:`int`=*`0`*, **`pickle_protocol`**:`int`=*`2`*) :: [`Callback`](/callback.core.html#Callback)\n",
|
|
"\n",
|
|
"[`Callback`](/callback.core.html#Callback) that returns all predictions and targets, optionally `with_input` or `with_loss`\n",
|
|
"\n",
|
|
"||Type|Default|Details|\n",
|
|
"|---|---|---|---|\n",
|
|
"|**`with_input`**|`bool`|`False`|Whether to return inputs|\n",
|
|
"|**`with_loss`**|`bool`|`False`|Whether to return losses|\n",
|
|
"|**`save_preds`**|`(str, PathLike)`|`None`|Path to save predictions|\n",
|
|
"|**`save_targs`**|`(str, PathLike)`|`None`|Path to save targets|\n",
|
|
"|**`with_preds`**|`bool`|`True`|Whether to return predictions|\n",
|
|
"|**`with_targs`**|`bool`|`True`|Whether to return targets|\n",
|
|
"|**`concat_dim`**|`int`|`0`|Dimension to concatenate returned tensors|\n",
|
|
"|**`pickle_protocol`**|`int`|`2`|Pickle protocol used to save predictions and targets|\n"
|
|
],
|
|
"text/plain": [
|
|
"<IPython.core.display.Markdown object>"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
}
|
|
],
|
|
"source": [
|
|
"show_doc(GatherPredsCallback, title_level=3)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "1c76da8f",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| export\n",
|
|
"class FetchPredsCallback(Callback):\n",
|
|
" \"A callback to fetch predictions during the training loop\"\n",
|
|
" remove_on_fetch = True\n",
|
|
" def __init__(self,\n",
|
|
" ds_idx:int=1, # Index of dataset, 0 for train, 1 for valid, used if `dl` is not present\n",
|
|
" dl:DataLoader=None, # `DataLoader` used for fetching `Learner` predictions\n",
|
|
" with_input:bool=False, # Whether to return inputs in `GatherPredsCallback`\n",
|
|
" with_decoded:bool=False, # Whether to return decoded predictions\n",
|
|
" cbs:Callback|MutableSequence=None, # `Callback` to temporarily remove from `Learner`\n",
|
|
" reorder:bool=True # Whether to sort prediction results\n",
|
|
" ):\n",
|
|
" self.cbs = L(cbs)\n",
|
|
" store_attr('ds_idx,dl,with_input,with_decoded,reorder')\n",
|
|
"\n",
|
|
" def after_validate(self):\n",
|
|
" \"Fetch predictions from `Learner` without `self.cbs` and `remove_on_fetch` callbacks\"\n",
|
|
" to_rm = L(cb for cb in self.learn.cbs if getattr(cb, 'remove_on_fetch', False))\n",
|
|
" with self.learn.removed_cbs(to_rm + self.cbs) as learn:\n",
|
|
" self.preds = learn.get_preds(ds_idx=self.ds_idx, dl=self.dl,\n",
|
|
" with_input=self.with_input, with_decoded=self.with_decoded, inner=True, reorder=self.reorder)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "f665bc5a",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/markdown": [
|
|
"<h3 id=\"FetchPredsCallback\" class=\"doc_header\"><code>class</code> <code>FetchPredsCallback</code><a href=\"\" class=\"source_link\" style=\"float:right\">[source]</a></h3>\n",
|
|
"\n",
|
|
"> <code>FetchPredsCallback</code>(**`ds_idx`**:`int`=*`1`*, **`dl`**:[`DataLoader`](/data.load.html#DataLoader)=*`None`*, **`with_input`**:`bool`=*`False`*, **`with_decoded`**:`bool`=*`False`*, **`cbs`**:`list`=*`None`*, **`reorder`**:`bool`=*`True`*) :: [`Callback`](/callback.core.html#Callback)\n",
|
|
"\n",
|
|
"A callback to fetch predictions during the training loop\n",
|
|
"\n",
|
|
"||Type|Default|Details|\n",
|
|
"|---|---|---|---|\n",
|
|
"|**`ds_idx`**|`int`|`1`|Index of dataset, 0 for train, 1 for valid, used if `dl` is not present|\n",
|
|
"|**`dl`**|`DataLoader`|`None`|`DataLoader` used for fetching `Learner` predictions|\n",
|
|
"|**`with_input`**|`bool`|`False`|Whether to return inputs in `GatherPredsCallback`|\n",
|
|
"|**`with_decoded`**|`bool`|`False`|Whether to return predicted classes|\n",
|
|
"|**`cbs`**|`list`|`None`|`Callback` list to add to the `Learner`|\n",
|
|
"|**`reorder`**|`bool`|`True`|Whether to sort prediction results|\n"
|
|
],
|
|
"text/plain": [
|
|
"<IPython.core.display.Markdown object>"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
}
|
|
],
|
|
"source": [
|
|
"show_doc(FetchPredsCallback, title_level=3)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "4ae886da",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Export -"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "f6bed810",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#| hide\n",
|
|
"from nbdev import nbdev_export\n",
|
|
"nbdev_export()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "120106bc",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": []
|
|
}
|
|
],
|
|
"metadata": {
|
|
"jupytext": {
|
|
"split_at_heading": true
|
|
},
|
|
"kernelspec": {
|
|
"display_name": "python3",
|
|
"language": "python",
|
|
"name": "python3"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|