chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,33 @@
|
||||
load("//bazel:python.bzl", "py_test_run_all_notebooks")
|
||||
|
||||
filegroup(
|
||||
name = "lightning_examples",
|
||||
srcs = glob(["*.ipynb"]),
|
||||
visibility = ["//doc:__subpackages__"],
|
||||
)
|
||||
|
||||
# GPU tests
|
||||
py_test_run_all_notebooks(
|
||||
size = "large",
|
||||
include = ["*.ipynb"],
|
||||
data = ["//doc/source/train/examples/lightning:lightning_examples"],
|
||||
exclude = [
|
||||
"dolly_lightning_fsdp_finetuning.ipynb", # Release Test
|
||||
"vicuna_13b_lightning_deepspeed_finetune.ipynb", # Release Test
|
||||
],
|
||||
tags = [
|
||||
"exclusive",
|
||||
"gpu",
|
||||
"ray_air",
|
||||
"team:ml",
|
||||
],
|
||||
)
|
||||
|
||||
filegroup(
|
||||
name = "lightning_examples_ci_configs",
|
||||
srcs = glob([
|
||||
"**/ci/aws.yaml",
|
||||
"**/ci/gce.yaml",
|
||||
]),
|
||||
visibility = ["//doc/source/train/examples:__pkg__"],
|
||||
)
|
||||
@@ -0,0 +1,842 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Fine-tune `dolly-v2-7b` with Ray Train, PyTorch Lightning and FSDP\n",
|
||||
"\n",
|
||||
"<a id=\"try-anyscale-quickstart-dolly_lightning_fsdp_finetuning\" href=\"https://console.anyscale.com/register/ha?render_flow=ray&utm_source=ray_docs&utm_medium=docs&utm_campaign=dolly_lightning_fsdp_finetuning\">\n",
|
||||
" <img src=\"../../../_static/img/run-on-anyscale.svg\" alt=\"try-anyscale-quickstart\">\n",
|
||||
"</a>\n",
|
||||
"<br></br>"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"In this example, we demonstrate how to use Ray Train to fine-tune a [`dolly-v2-7b`](https://huggingface.co/databricks/dolly-v2-7b) model. `dolly-v2-7b` is a 7 billion parameter causal language model created by Databricks, derived from EleutherAI’s [Pythia-6.9b](https://huggingface.co/EleutherAI/pythia-6.9b), and fine-tuned on a [~15K record instruction corpus](https://github.com/databrickslabs/dolly/tree/master/data).\n",
|
||||
"\n",
|
||||
"We load the pre-trained model from the HuggingFace model hub into a LightningModule and launch an FSDP fine-tuning job across 16 T4 GPUs with the help of {class}`Ray TorchTrainer <ray.train.torch.TorchTrainer>`. It is also straightforward to fine-tune other similar large language models in a similar manner as shown in this example.\n",
|
||||
"\n",
|
||||
"Before starting this example, we highly recommend reading [Ray Train Key Concepts](train-key-concepts) and [Ray Data Quickstart](data_quickstart)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"In this , we demonstrate how to use Ray Train to fine-tune a [`dolly-v2-7b`](https://huggingface.co/databricks/dolly-v2-7b) model. `dolly-v2-7b` is a 7 billion parameter causal language model created by Databricks, derived from EleutherAI’s [Pythia-6.9b](https://huggingface.co/EleutherAI/pythia-6.9b), and fine-tuned on a [~15K record instruction corpus](https://github.com/databrickslabs/dolly/tree/master/data).\n",
|
||||
"\n",
|
||||
"We load the pre-trained model from the HuggingFace model hub into a LightningModule and launch an FSDP fine-tuning job across 16 T4 GPUs with the help of {class}`Ray TorchTrainer <ray.train.torch.TorchTrainer>`. It is also straightforward to fine-tune other similar large language models in a similar manner as shown in this example.\n",
|
||||
"\n",
|
||||
"Before starting this example, we highly recommend reading [Ray Train Key Concepts](train-key-concepts) and [Ray Data Quickstart](data_quickstart)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Set up ray cluster \n",
|
||||
"In this example, we are using a Ray cluster with a `m5.2xlarge` head node and 4 `g4dn.12xlarge` worker nodes. Each `g4dn.12xlarge has four Tesla T4 GPUs. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import ray\n",
|
||||
"ray.init()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We then install the necessary dependencies on each node:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%bash\n",
|
||||
"pip install datasets\n",
|
||||
"pip install evaluate\n",
|
||||
"pip install \"transformers>=4.26.0\"\n",
|
||||
"pip install \"torch>=1.12.0\"\n",
|
||||
"pip install \"lightning>=2.0\"\n",
|
||||
"pip install \"pydantic>=2,<3\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"MODEL_NAME = \"databricks/dolly-v2-7b\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Prepare your data \n",
|
||||
"We are using tiny_shakespeare for fine-tuning, which contains 40,000 lines of Shakespeare from a variety of Shakespeare's plays. Featured in Andrej Karpathy's blog post ['The Unreasonable Effectiveness of Recurrent Neural Networks'](http://karpathy.github.io/2015/05/21/rnn-effectiveness/). \n",
|
||||
"\n",
|
||||
"Dataset samples:\n",
|
||||
"```\n",
|
||||
"BAPTISTA:\n",
|
||||
"I know him well: you are welcome for his sake.\n",
|
||||
"\n",
|
||||
"GREMIO:\n",
|
||||
"Saving your tale, Petruchio, I pray,\n",
|
||||
"Let us, that are poor petitioners, speak too:\n",
|
||||
"Baccare! you are marvellous forward.\n",
|
||||
"\n",
|
||||
"PETRUCHIO:\n",
|
||||
"O, pardon me, Signior Gremio; I would fain be doing.\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
"Here, we have adopted similar pre-processing logic from another demo: {doc}`GPT-J-6B Fine-Tuning with Ray Train and DeepSpeed <../deepspeed/gptj_deepspeed_fine_tuning>`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import ray\n",
|
||||
"import pandas as pd\n",
|
||||
"from datasets import load_dataset\n",
|
||||
"from transformers import AutoTokenizer, AutoModelForCausalLM\n",
|
||||
"\n",
|
||||
"def split_text(batch: pd.DataFrame) -> pd.DataFrame:\n",
|
||||
" text = list(batch[\"text\"])\n",
|
||||
" flat_text = \"\".join(text)\n",
|
||||
" split_text = [\n",
|
||||
" x.strip()\n",
|
||||
" for x in flat_text.split(\"\\n\")\n",
|
||||
" if x.strip() and not x.strip()[-1] == \":\"\n",
|
||||
" ]\n",
|
||||
" return pd.DataFrame(split_text, columns=[\"text\"])\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def tokenize(batch: pd.DataFrame) -> dict:\n",
|
||||
" tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, padding_side=\"left\")\n",
|
||||
" tokenizer.pad_token = tokenizer.eos_token\n",
|
||||
" ret = tokenizer(\n",
|
||||
" list(batch[\"text\"]),\n",
|
||||
" truncation=True,\n",
|
||||
" max_length=256,\n",
|
||||
" padding=\"max_length\",\n",
|
||||
" return_tensors=\"np\",\n",
|
||||
" )\n",
|
||||
" ret[\"labels\"] = ret[\"input_ids\"].copy()\n",
|
||||
" return dict(ret)\n",
|
||||
"\n",
|
||||
"hf_dataset = load_dataset(\"tiny_shakespeare\", trust_remote_code=True)\n",
|
||||
"train_ds = ray.data.from_huggingface(hf_dataset[\"train\"])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We first split the original paragraphs into multiple sentences, then tokenize them. Here are some samples:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# First split the dataset into multiple sentences.\n",
|
||||
"train_ds = train_ds.map_batches(split_text, batch_format=\"pandas\", batch_size=\"auto\")\n",
|
||||
"train_ds.take(10)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Then tokenize the dataset.\n",
|
||||
"train_ds = train_ds.map_batches(tokenize, batch_format=\"pandas\", batch_size=\"auto\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Define your lightning model\n",
|
||||
"\n",
|
||||
"In this example, we use the [dolly-v2-7b](https://huggingface.co/databricks/dolly-v2-7b) model for finetuning. It is an instruction-following large language model trained on the Databricks machine learning platform that is licensed for commercial use. We load the model weights from Huggingface Model Hub and encapsulate it into a `pl.LightningModule`.\n",
|
||||
"\n",
|
||||
":::{note}\n",
|
||||
"Make sure you pass the FSDP wrapped model parameters `self.trainer.model.parameters()` into the optimizer, instead of `self.model.parameters()`. \n",
|
||||
":::\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import torch\n",
|
||||
"import lightning.pytorch as pl\n",
|
||||
"\n",
|
||||
"class DollyV2Model(pl.LightningModule):\n",
|
||||
" def __init__(self, lr=2e-5, eps=1e-8):\n",
|
||||
" super().__init__()\n",
|
||||
" self.save_hyperparameters()\n",
|
||||
" self.lr = lr\n",
|
||||
" self.eps = eps\n",
|
||||
" self.model = AutoModelForCausalLM.from_pretrained(MODEL_NAME)\n",
|
||||
"\n",
|
||||
" def forward(self, batch):\n",
|
||||
" outputs = self.model(\n",
|
||||
" batch[\"input_ids\"], \n",
|
||||
" attention_mask=batch[\"attention_mask\"], \n",
|
||||
" labels=batch[\"labels\"]\n",
|
||||
" )\n",
|
||||
" return outputs.loss\n",
|
||||
"\n",
|
||||
" def training_step(self, batch, batch_idx):\n",
|
||||
" loss = self.forward(batch)\n",
|
||||
" self.log(\"train_loss\", loss, prog_bar=True, on_step=True)\n",
|
||||
" return loss\n",
|
||||
"\n",
|
||||
" def configure_optimizers(self):\n",
|
||||
" if self.global_rank == 0:\n",
|
||||
" print(self.trainer.model)\n",
|
||||
" return torch.optim.AdamW(self.trainer.model.parameters(), lr=self.lr, eps=self.eps)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Configure your FSDP strategy\n",
|
||||
"As `dolly-v2-7b` is a relatively large model, it cannot be properly fit into a single commercial GPU. In this example, we use the FSDP strategy to shard model parameters across multiple workers. This allows us to avoid GPU out-of-memory issues and support a larger global batch size.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Image source: [Fully Sharded Data Parallel: faster AI training with fewer GPUs](https://engineering.fb.com/2021/07/15/open-source/fsdp/)\n",
|
||||
"\n",
|
||||
":::{note}\n",
|
||||
"FSDP is a type of data parallelism that shards model parameters, optimizer states and gradients across DDP ranks. This was inspired by Xu et al. as well as the ZeRO Stage 3 from DeepSpeed. You may refer to these blogs for more information:\n",
|
||||
"\n",
|
||||
"- [Fully Sharded Data Parallel: faster AI training with fewer GPUs](https://engineering.fb.com/2021/07/15/open-source/fsdp/)\n",
|
||||
"- [Getting Started with Fully Sharded Data Parallel(FSDP)](https://pytorch.org/tutorials/intermediate/FSDP_tutorial.html#:~:text=FSDP%20is%20a%20type%20of,sizes%20for%20our%20training%20job.)\n",
|
||||
"- [PyTorch FSDP Tutorial](https://www.youtube.com/watch?v=8_k76AHu__s&list=PL_lsbAsL_o2BT6aerEKgIoufVD_fodnuT)\n",
|
||||
":::\n",
|
||||
"\n",
|
||||
"To start training with Lightning's [FSDPStrategy](https://lightning.ai/docs/pytorch/stable/api/lightning.pytorch.strategies.FSDPStrategy.html#lightning.pytorch.strategies.FSDPStrategy), you only need to create a {class}`~ray.train.lightning.RayFSDPStrategy` with the same initialization arguments.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import functools\n",
|
||||
"import lightning.pytorch as pl \n",
|
||||
"\n",
|
||||
"from torch.distributed.fsdp.wrap import transformer_auto_wrap_policy\n",
|
||||
"from torch.distributed.fsdp import ShardingStrategy, BackwardPrefetch\n",
|
||||
"from transformers.models.gpt_neox.modeling_gpt_neox import GPTNeoXLayer\n",
|
||||
"\n",
|
||||
"from ray.train.lightning import RayFSDPStrategy\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# Define the model sharding policy:\n",
|
||||
"# Wrap every GPTNeoXLayer as its own FSDP instance\n",
|
||||
"auto_wrap_policy = functools.partial(\n",
|
||||
" transformer_auto_wrap_policy,\n",
|
||||
" transformer_layer_cls = {GPTNeoXLayer}\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"fsdp_strategy = RayFSDPStrategy(\n",
|
||||
" sharding_strategy=ShardingStrategy.FULL_SHARD,\n",
|
||||
" backward_prefetch=BackwardPrefetch.BACKWARD_PRE,\n",
|
||||
" forward_prefetch=True,\n",
|
||||
" auto_wrap_policy=auto_wrap_policy,\n",
|
||||
" limit_all_gathers=True,\n",
|
||||
" activation_checkpointing=[GPTNeoXLayer],\n",
|
||||
" cpu_offload=True\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
":::{tip}\n",
|
||||
"\n",
|
||||
"Some tips for FSDP configuration:\n",
|
||||
"- `sharding_strategy`:\n",
|
||||
" - `ShardingStrategy.NO_SHARD`: Parameters, gradients, and optimizer states are not sharded. Similar to DDP.\n",
|
||||
" - `ShardingStrategy.SHARD_GRAD_OP`: Gradients and optimizer states are sharded during computation, and additionally, parameters are sharded outside computation. Similar to ZeRO stage-2.\n",
|
||||
" - `ShardingStrategy.FULL_SHARD`: Parameters, gradients, and optimizer states are sharded. It has minimal GRAM usage among the 3 options. Similar to ZeRO stage-3.\n",
|
||||
"- `auto_wrap_policy`:\n",
|
||||
" - Model layers are often wrapped with FSDP in a layered fashion. This means that only the layers in a single FSDP instance are required to aggregate all parameters to a single device during forwarding or backward calculations.\n",
|
||||
" - Use `transformer_auto_wrap_policy` to automatically wrap each Transformer Block into a single FSDP instance. \n",
|
||||
"- `backward_prefetch` and `forward_prefetch`:\n",
|
||||
" - Overlap the upcoming all-gather while executing the current forward/backward pass. It can improve throughput but may slightly increase peak memory usage.\n",
|
||||
":::"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Fine-tune with Ray TorchTrainer\n",
|
||||
"\n",
|
||||
"Ray TorchTrainer allows you to scale your PyTorch Lightning training workload over multiple nodes. See {ref}`train_scaling_config` for more details."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"num_workers = 16\n",
|
||||
"batch_size_per_worker = 5"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"tags": [
|
||||
"remove-cell"
|
||||
]
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# To accelerate release tests\n",
|
||||
"train_ds = train_ds.limit(num_workers * batch_size_per_worker * 5) # each worker has 5 batches"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Additionally, remember to define a Lightning callback that saves and reports checkpoints. Ray Train offers a simple implementation, {meth}`~ray.train.lightning.RayTrainReportCallback`, which persists your checkpoint and metrics in remote storage at the end of each training epoch. \n",
|
||||
"\n",
|
||||
"Note you can also implement your own report callback with customized logics, such as saving customized checkpoint files or reporting at a different frequency."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"tags": [
|
||||
"remove-cell"
|
||||
]
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from lightning.pytorch.callbacks import TQDMProgressBar\n",
|
||||
"\n",
|
||||
"# Create a customized progress bar for Ray Data Iterable Dataset\n",
|
||||
"class DollyV2ProgressBar(TQDMProgressBar):\n",
|
||||
" def __init__(self, num_iters_per_epoch, *args, **kwargs):\n",
|
||||
" super().__init__(*args, **kwargs)\n",
|
||||
" self.num_iters_per_epoch = num_iters_per_epoch\n",
|
||||
" \n",
|
||||
" def on_train_epoch_start(self, trainer, *_):\n",
|
||||
" super().on_train_epoch_start(trainer, *_)\n",
|
||||
" self.train_progress_bar.reset(self.num_iters_per_epoch)\n",
|
||||
"\n",
|
||||
"num_iters_per_epoch = train_ds.count() // (num_workers * batch_size_per_worker)\n",
|
||||
"prog_bar = DollyV2ProgressBar(num_iters_per_epoch)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 13,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from ray.train import Checkpoint\n",
|
||||
"from ray.train.lightning import RayLightningEnvironment, RayTrainReportCallback, prepare_trainer\n",
|
||||
"\n",
|
||||
"# Training function for each worker\n",
|
||||
"def train_func(config):\n",
|
||||
" lr = config[\"lr\"]\n",
|
||||
" eps = config[\"eps\"]\n",
|
||||
" strategy = config[\"strategy\"]\n",
|
||||
" batch_size_per_worker = config[\"batch_size_per_worker\"]\n",
|
||||
"\n",
|
||||
" # Model\n",
|
||||
" model = DollyV2Model(lr=lr, eps=eps)\n",
|
||||
"\n",
|
||||
" # Ray Data Ingestion\n",
|
||||
" train_ds = ray.train.get_dataset_shard(\"train\")\n",
|
||||
" train_dataloader = train_ds.iter_torch_batches(batch_size=batch_size_per_worker)\n",
|
||||
"\n",
|
||||
" # Lightning Trainer\n",
|
||||
" trainer = pl.Trainer(\n",
|
||||
" max_epochs=1, \n",
|
||||
" devices=\"auto\",\n",
|
||||
" accelerator=\"auto\", \n",
|
||||
" precision=\"16-mixed\",\n",
|
||||
" strategy=strategy,\n",
|
||||
" plugins=[RayLightningEnvironment()],\n",
|
||||
" callbacks=[RayTrainReportCallback()],\n",
|
||||
" enable_checkpointing=False,\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" trainer = prepare_trainer(trainer)\n",
|
||||
"\n",
|
||||
" trainer.fit(model, train_dataloaders=train_dataloader)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"```{note}\n",
|
||||
"Since this example runs with multiple nodes, we need to persist checkpoints\n",
|
||||
"and other outputs to some external storage for access after training has completed.\n",
|
||||
"**You should set up cloud storage or NFS, then replace `storage_path` with your own cloud bucket URI or NFS path.**\n",
|
||||
"\n",
|
||||
"See the [storage guide](tune-storage-options) for more details.\n",
|
||||
"```"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 14,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"storage_path=\"s3://your-bucket-here\" # TODO: Set up cloud storage\n",
|
||||
"# storage_path=\"/mnt/path/to/nfs\" # TODO: Alternatively, set up NFS"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 15,
|
||||
"metadata": {
|
||||
"tags": [
|
||||
"remove-cell"
|
||||
]
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"storage_path = \"/mnt/cluster_storage\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 16,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"\u001b[36m(TrainController pid=6878)\u001b[0m /home/ray/anaconda3/lib/python3.10/site-packages/transformers/utils/generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.\n",
|
||||
"\u001b[36m(TrainController pid=6878)\u001b[0m _torch_pytree._register_pytree_node(\n",
|
||||
"\u001b[36m(TrainController pid=6878)\u001b[0m /home/ray/anaconda3/lib/python3.10/site-packages/transformers/utils/generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.\n",
|
||||
"\u001b[36m(TrainController pid=6878)\u001b[0m _torch_pytree._register_pytree_node(\n",
|
||||
"\u001b[36m(TrainController pid=6878)\u001b[0m [State Transition] INITIALIZING -> SCHEDULING.\n",
|
||||
"\u001b[36m(TrainController pid=6878)\u001b[0m Attempting to start training worker group of size 16 with the following resources: [{'GPU': 1}] * 16\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4349, ip=10.0.157.249)\u001b[0m /home/ray/anaconda3/lib/python3.10/site-packages/transformers/utils/generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4349, ip=10.0.157.249)\u001b[0m _torch_pytree._register_pytree_node(\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4350, ip=10.0.157.249)\u001b[0m /home/ray/anaconda3/lib/python3.10/site-packages/transformers/utils/generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4350, ip=10.0.157.249)\u001b[0m _torch_pytree._register_pytree_node(\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m Setting up process group for: env:// [rank=0, world_size=16]\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4349, ip=10.0.157.249)\u001b[0m /home/ray/anaconda3/lib/python3.10/site-packages/huggingface_hub/file_download.py:795: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4349, ip=10.0.157.249)\u001b[0m warnings.warn(\n",
|
||||
"\u001b[36m(TrainController pid=6878)\u001b[0m Started training worker group of size 16: \n",
|
||||
"\u001b[36m(TrainController pid=6878)\u001b[0m - (ip=10.0.163.141, pid=4088) world_rank=0, local_rank=0, node_rank=0\n",
|
||||
"\u001b[36m(TrainController pid=6878)\u001b[0m - (ip=10.0.163.141, pid=4089) world_rank=1, local_rank=1, node_rank=0\n",
|
||||
"\u001b[36m(TrainController pid=6878)\u001b[0m - (ip=10.0.163.141, pid=4090) world_rank=2, local_rank=2, node_rank=0\n",
|
||||
"\u001b[36m(TrainController pid=6878)\u001b[0m - (ip=10.0.163.141, pid=4091) world_rank=3, local_rank=3, node_rank=0\n",
|
||||
"\u001b[36m(TrainController pid=6878)\u001b[0m - (ip=10.0.166.248, pid=4338) world_rank=4, local_rank=0, node_rank=1\n",
|
||||
"\u001b[36m(TrainController pid=6878)\u001b[0m - (ip=10.0.166.248, pid=4337) world_rank=5, local_rank=1, node_rank=1\n",
|
||||
"\u001b[36m(TrainController pid=6878)\u001b[0m - (ip=10.0.166.248, pid=4340) world_rank=6, local_rank=2, node_rank=1\n",
|
||||
"\u001b[36m(TrainController pid=6878)\u001b[0m - (ip=10.0.166.248, pid=4339) world_rank=7, local_rank=3, node_rank=1\n",
|
||||
"\u001b[36m(TrainController pid=6878)\u001b[0m - (ip=10.0.191.43, pid=4090) world_rank=8, local_rank=0, node_rank=2\n",
|
||||
"\u001b[36m(TrainController pid=6878)\u001b[0m - (ip=10.0.191.43, pid=4248) world_rank=9, local_rank=1, node_rank=2\n",
|
||||
"\u001b[36m(TrainController pid=6878)\u001b[0m - (ip=10.0.191.43, pid=4246) world_rank=10, local_rank=2, node_rank=2\n",
|
||||
"\u001b[36m(TrainController pid=6878)\u001b[0m - (ip=10.0.191.43, pid=4247) world_rank=11, local_rank=3, node_rank=2\n",
|
||||
"\u001b[36m(TrainController pid=6878)\u001b[0m - (ip=10.0.157.249, pid=4349) world_rank=12, local_rank=0, node_rank=3\n",
|
||||
"\u001b[36m(TrainController pid=6878)\u001b[0m - (ip=10.0.157.249, pid=4350) world_rank=13, local_rank=1, node_rank=3\n",
|
||||
"\u001b[36m(TrainController pid=6878)\u001b[0m - (ip=10.0.157.249, pid=4347) world_rank=14, local_rank=2, node_rank=3\n",
|
||||
"\u001b[36m(TrainController pid=6878)\u001b[0m - (ip=10.0.157.249, pid=4348) world_rank=15, local_rank=3, node_rank=3\n",
|
||||
"\u001b[36m(TrainController pid=6878)\u001b[0m [State Transition] SCHEDULING -> RUNNING.\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4246, ip=10.0.191.43)\u001b[0m /home/ray/anaconda3/lib/python3.10/site-packages/transformers/utils/generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.\u001b[32m [repeated 31x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)\u001b[0m\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4246, ip=10.0.191.43)\u001b[0m _torch_pytree._register_pytree_node(\u001b[32m [repeated 31x across cluster]\u001b[0m\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"\u001b[33m(raylet)\u001b[0m WARNING: 4 PYTHON worker processes have been started on node: 3ffdd02cd1be6b69a64a97e08f75fc0a80eddcf0caa627f3f4266c95 with address: 10.0.150.21. This could be a result of using a large number of actors, or due to tasks blocked in ray.get() calls (see https://github.com/ray-project/ray/issues/3644 for some discussion of workarounds).\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m GPU available: True (cuda), used: True\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m TPU available: False, using: 0 TPU cores\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m HPU available: False, using: 0 HPUs\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4246, ip=10.0.191.43)\u001b[0m /home/ray/anaconda3/lib/python3.10/site-packages/huggingface_hub/file_download.py:795: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\u001b[32m [repeated 15x across cluster]\u001b[0m\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4246, ip=10.0.191.43)\u001b[0m warnings.warn(\u001b[32m [repeated 15x across cluster]\u001b[0m\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m 2025-09-30 14:20:07.970624: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m 2025-09-30 14:20:08.208262: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m 2025-09-30 14:20:08.208291: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m 2025-09-30 14:20:08.231782: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m 2025-09-30 14:20:08.277889: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
|
||||
"\u001b[36m(SplitCoordinator pid=7661)\u001b[0m /home/ray/anaconda3/lib/python3.10/site-packages/transformers/utils/generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.\n",
|
||||
"\u001b[36m(SplitCoordinator pid=7661)\u001b[0m _torch_pytree._register_pytree_node(\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m 2025-09-30 14:20:10.134645: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3]\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m FullyShardedDataParallel(\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m (_fsdp_wrapped_module): DollyV2Model(\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m (model): GPTNeoXForCausalLM(\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m (gpt_neox): GPTNeoXModel(\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m (embed_in): Embedding(50280, 4096)\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m (emb_dropout): Dropout(p=0.0, inplace=False)\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m (layers): ModuleList(\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m (0-31): 32 x FullyShardedDataParallel(\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m (_fsdp_wrapped_module): CheckpointWrapper(\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m (_checkpoint_wrapped_module): GPTNeoXLayer(\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m (input_layernorm): LayerNorm((4096,), eps=1e-05, elementwise_affine=True)\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m (post_attention_layernorm): LayerNorm((4096,), eps=1e-05, elementwise_affine=True)\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m (post_attention_dropout): Dropout(p=0.0, inplace=False)\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m (post_mlp_dropout): Dropout(p=0.0, inplace=False)\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m (attention): GPTNeoXAttention(\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m (rotary_emb): GPTNeoXRotaryEmbedding()\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m (query_key_value): Linear(in_features=4096, out_features=12288, bias=True)\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m (dense): Linear(in_features=4096, out_features=4096, bias=True)\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m (attention_dropout): Dropout(p=0.0, inplace=False)\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m )\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m (mlp): GPTNeoXMLP(\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m (dense_h_to_4h): Linear(in_features=4096, out_features=16384, bias=True)\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m (dense_4h_to_h): Linear(in_features=16384, out_features=4096, bias=True)\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m (act): GELUActivation()\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m )\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m )\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m )\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m )\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m )\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m (final_layer_norm): LayerNorm((4096,), eps=1e-05, elementwise_affine=True)\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m )\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m (embed_out): Linear(in_features=4096, out_features=50280, bias=False)\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m )\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m )\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m )\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m /tmp/ray/session_2025-09-30_14-10-46_627006_2395/runtime_resources/pip/72a6e451f55d87eb50ebbf5bc30a4a57ed513d34/virtualenv/lib/python3.10/site-packages/lightning/pytorch/utilities/model_summary/model_summary.py:231: Precision 16-mixed is not supported by the model summary. Estimated model size in MB will not be accurate. Using 32 bits instead.\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m \n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m | Name | Type | Params | Mode\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m ----------------------------------------------------\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m 0 | model | GPTNeoXForCausalLM | 428 M | eval\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m ----------------------------------------------------\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m 428 M Trainable params\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m 0 Non-trainable params\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m 428 M Total params\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m 1,714.014 Total estimated model params size (MB)\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m 64 Modules in train mode\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m 455 Modules in eval mode\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4246, ip=10.0.191.43)\u001b[0m LOCAL_RANK: 2 - CUDA_VISIBLE_DEVICES: [0,1,2,3]\u001b[32m [repeated 15x across cluster]\u001b[0m\n",
|
||||
"\u001b[36m(SplitCoordinator pid=7661)\u001b[0m Registered dataset logger for dataset train_12_0\n",
|
||||
"\u001b[36m(SplitCoordinator pid=7661)\u001b[0m Starting execution of Dataset train_12_0. Full logs are in /tmp/ray/session_2025-09-30_14-10-46_627006_2395/logs/ray-data\n",
|
||||
"\u001b[36m(SplitCoordinator pid=7661)\u001b[0m Execution plan of Dataset train_12_0: InputDataBuffer[Input] -> TaskPoolMapOperator[MapBatches(split_text)->MapBatches(tokenize)] -> LimitOperator[limit=400] -> OutputSplitter[split(16, equal=True)]\n",
|
||||
"\u001b[36m(SplitCoordinator pid=7661)\u001b[0m ⚠️ Ray's object store is configured to use only 28.7% of available memory (229.3GiB out of 800.0GiB total). For optimal Ray Data performance, we recommend setting the object store to at least 50% of available memory. You can do this by setting the 'object_store_memory' parameter when calling ray.init() or by setting the RAY_DEFAULT_OBJECT_STORE_MEMORY_PROPORTION environment variable.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"application/vnd.jupyter.widget-view+json": {
|
||||
"model_id": "ab803bffe2224e6591bc452cac07f2a8",
|
||||
"version_major": 2,
|
||||
"version_minor": 0
|
||||
},
|
||||
"text/plain": [
|
||||
"(pid=7661) Running 0: 0.00 row [00:00, ? row/s]"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"application/vnd.jupyter.widget-view+json": {
|
||||
"model_id": "8d647925625f49c19e16fddce0fab359",
|
||||
"version_major": 2,
|
||||
"version_minor": 0
|
||||
},
|
||||
"text/plain": [
|
||||
"(pid=7661) - MapBatches(split_text)->MapBatches(tokenize) 1: 0.00 row [00:00, ? row/s]"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"application/vnd.jupyter.widget-view+json": {
|
||||
"model_id": "30978a8d911a44dcb168b0d5a386a42c",
|
||||
"version_major": 2,
|
||||
"version_minor": 0
|
||||
},
|
||||
"text/plain": [
|
||||
"(pid=7661) - limit=400 2: 0.00 row [00:00, ? row/s]"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"application/vnd.jupyter.widget-view+json": {
|
||||
"model_id": "432c8cb6d3e84b749d9341ff104bb25c",
|
||||
"version_major": 2,
|
||||
"version_minor": 0
|
||||
},
|
||||
"text/plain": [
|
||||
"(pid=7661) - split(16, equal=True) 3: 0.00 row [00:00, ? row/s]"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
},
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"\u001b[36m(MapBatches(split_text)->MapBatches(tokenize) pid=4089, ip=10.0.191.43)\u001b[0m /home/ray/anaconda3/lib/python3.10/site-packages/transformers/utils/generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.\n",
|
||||
"\u001b[36m(MapBatches(split_text)->MapBatches(tokenize) pid=4089, ip=10.0.191.43)\u001b[0m _torch_pytree._register_pytree_node(\n",
|
||||
"\u001b[36m(MapBatches(split_text)->MapBatches(tokenize) pid=4089, ip=10.0.191.43)\u001b[0m /home/ray/anaconda3/lib/python3.10/site-packages/huggingface_hub/file_download.py:795: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n",
|
||||
"\u001b[36m(MapBatches(split_text)->MapBatches(tokenize) pid=4089, ip=10.0.191.43)\u001b[0m warnings.warn(\n",
|
||||
"\u001b[36m(MapBatches(split_text)->MapBatches(tokenize) pid=4089, ip=10.0.191.43)\u001b[0m Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n",
|
||||
"\u001b[36m(SplitCoordinator pid=7661)\u001b[0m /home/ray/anaconda3/lib/python3.10/site-packages/ray/anyscale/data/_internal/cluster_autoscaler/productivity_calculator.py:174: RuntimeWarning: invalid value encountered in divide\n",
|
||||
"\u001b[36m(SplitCoordinator pid=7661)\u001b[0m gpu_fraction_per_op = (optimal_num_tasks_per_op * num_gpus_per_op) / np.sum(\n",
|
||||
"\u001b[36m(SplitCoordinator pid=7661)\u001b[0m ✔️ Dataset train_12_0 execution finished in 5.10 seconds\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m /tmp/ray/session_2025-09-30_14-10-46_627006_2395/runtime_resources/pip/72a6e451f55d87eb50ebbf5bc30a4a57ed513d34/virtualenv/lib/python3.10/site-packages/lightning/pytorch/loops/fit_loop.py:527: Found 455 module(s) in eval mode at the start of training. This may lead to unexpected behavior during training. If this is intentional, you can ignore this warning.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Epoch 0: | | 0/? [00:00<?, ?it/s] 141)\u001b[0m \n",
|
||||
"Epoch 0: | | 1/? [00:20<00:00, 0.05it/s, v_num=0, train_loss=14.60]\n",
|
||||
"Epoch 0: | | 2/? [00:36<00:00, 0.06it/s, v_num=0, train_loss=13.90]\n",
|
||||
"Epoch 0: | | 3/? [00:52<00:00, 0.06it/s, v_num=0, train_loss=14.80]\n",
|
||||
"Epoch 0: | | 4/? [01:09<00:00, 0.06it/s, v_num=0, train_loss=14.10]\n",
|
||||
"Epoch 0: | | 5/? [01:27<00:00, 0.06it/s, v_num=0, train_loss=14.90]\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"\u001b[36m(RayTrainWorker pid=4090, ip=10.0.191.43)\u001b[0m Checkpoint successfully created at: Checkpoint(filesystem=local, path=/mnt/cluster_storage/finetune_dolly-v2-7b-trial1/checkpoint_2025-09-30_14-23-45.998076)\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4090, ip=10.0.191.43)\u001b[0m Reporting training result 1: TrainingResult(checkpoint=Checkpoint(filesystem=local, path=/mnt/cluster_storage/finetune_dolly-v2-7b-trial1/checkpoint_2025-09-30_14-23-45.998076), metrics={'train_loss': 14.9804105758667, 'epoch': 0, 'step': 5})\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m Checkpoint successfully created at: Checkpoint(filesystem=local, path=/mnt/cluster_storage/finetune_dolly-v2-7b-trial1/checkpoint_2025-09-30_14-23-45.998076)\u001b[32m [repeated 12x across cluster]\u001b[0m\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m Reporting training result 1: TrainingResult(checkpoint=Checkpoint(filesystem=local, path=/mnt/cluster_storage/finetune_dolly-v2-7b-trial1/checkpoint_2025-09-30_14-23-45.998076), metrics={'train_loss': 14.879826545715332, 'epoch': 0, 'step': 5})\u001b[32m [repeated 12x across cluster]\u001b[0m\n",
|
||||
"\u001b[36m(RayTrainWorker pid=4088, ip=10.0.163.141)\u001b[0m `Trainer.fit` stopped: `max_epochs=1` reached.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Epoch 0: | | 5/? [06:07<00:00, 0.01it/s, v_num=0, train_loss=14.90]\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"Result(metrics={'train_loss': 14.879826545715332, 'epoch': 0, 'step': 5}, checkpoint=Checkpoint(filesystem=local, path=/mnt/cluster_storage/finetune_dolly-v2-7b-trial1/checkpoint_2025-09-30_14-23-45.998076), error=None, path='/mnt/cluster_storage/finetune_dolly-v2-7b-trial1', metrics_dataframe= train_loss epoch step\n",
|
||||
"0 14.879827 0 5, best_checkpoints=[(Checkpoint(filesystem=local, path=/mnt/cluster_storage/finetune_dolly-v2-7b-trial1/checkpoint_2025-09-30_14-23-45.998076), {'train_loss': 14.879826545715332, 'epoch': 0, 'step': 5})], _storage_filesystem=<pyarrow._fs.LocalFileSystem object at 0x737af20de5b0>)"
|
||||
]
|
||||
},
|
||||
"execution_count": 16,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from ray.train.torch import TorchTrainer\n",
|
||||
"from ray.train import RunConfig, ScalingConfig, CheckpointConfig\n",
|
||||
"\n",
|
||||
"# Save Ray Train checkpoints according to the performance on validation set\n",
|
||||
"run_config = RunConfig(\n",
|
||||
" name=\"finetune_dolly-v2-7b-trial1\",\n",
|
||||
" storage_path=storage_path,\n",
|
||||
" checkpoint_config=CheckpointConfig(num_to_keep=1),\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"# Scale the FSDP training workload across 16 GPUs\n",
|
||||
"# You can change this config based on your compute resources.\n",
|
||||
"scaling_config = ScalingConfig(\n",
|
||||
" num_workers=num_workers, use_gpu=True\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"# Configuration to pass into train_func\n",
|
||||
"train_config = {\n",
|
||||
" \"lr\": 2e-5,\n",
|
||||
" \"eps\": 1e-8,\n",
|
||||
" \"strategy\": fsdp_strategy,\n",
|
||||
" \"batch_size_per_worker\": batch_size_per_worker\n",
|
||||
"}\n",
|
||||
"\n",
|
||||
"# Define a TorchTrainer and launch you training workload\n",
|
||||
"ray_trainer = TorchTrainer(\n",
|
||||
" train_func,\n",
|
||||
" train_loop_config=train_config,\n",
|
||||
" run_config=run_config,\n",
|
||||
" scaling_config=scaling_config,\n",
|
||||
" datasets={\"train\": train_ds},\n",
|
||||
")\n",
|
||||
"result = ray_trainer.fit()\n",
|
||||
"\n",
|
||||
"result\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We finished training in 2877s. The price for an on-demand g4dn.4xlarge instance is `$1.204/hour`, while a g4dn.8xlarge instance costs `$2.176/hour`. The total cost would be `($1.204 * 15 + $2.176) * 2877 / 3600 = $16.17`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Text-generation with HuggingFace Pipeline\n",
|
||||
"\n",
|
||||
"We can use the [HuggingFace Pipeline](https://huggingface.co/docs/transformers/main_classes/pipelines) to generate predictions from our fine-tuned model. Let's input some prompts and see if our tuned Dolly can speak like Shakespeare:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"from transformers import pipeline\n",
|
||||
"\n",
|
||||
"@ray.remote(num_gpus=1)\n",
|
||||
"def generate_tokens():\n",
|
||||
" tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, padding_side=\"right\")\n",
|
||||
"\n",
|
||||
" ckpt_path = os.path.join(result.checkpoint.path, \"checkpoint.ckpt\")\n",
|
||||
"\n",
|
||||
" dolly = DollyV2Model.load_from_checkpoint(ckpt_path, map_location=torch.device(\"cpu\"))\n",
|
||||
"\n",
|
||||
" nlp_pipeline = pipeline(\n",
|
||||
" task=\"text-generation\", \n",
|
||||
" model=dolly.model, \n",
|
||||
" tokenizer=tokenizer, \n",
|
||||
" device_map=\"auto\"\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" tokens = []\n",
|
||||
" for prompt in [\"This is\", \"I am\", \"Once more\"]:\n",
|
||||
" tokens.append(nlp_pipeline(prompt, max_new_tokens=20, do_sample=True, pad_token_id=tokenizer.eos_token_id))\n",
|
||||
"\n",
|
||||
" return tokens\n",
|
||||
"\n",
|
||||
"ref = generate_tokens.remote()\n",
|
||||
"output = ray.get(ref)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 23,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"[{'generated_text': \"This is more like it:\\n\\nIt's just a guess, but maybe the extra processing power of Intel\"}]\n",
|
||||
"[{'generated_text': \"I am the biggest fan of your wife's writing, and this novella was fantastic. So interesting to see\"}]\n",
|
||||
"[{'generated_text': 'Once more I wish I could make sense of it.\" \"My friend, you can leave all this behind you'}]\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"for generated_tokens in output:\n",
|
||||
" print(generated_tokens)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"References:\n",
|
||||
"- [PyTorch FSDP Tutorial](https://www.youtube.com/watch?v=8_k76AHu__s&list=PL_lsbAsL_o2BT6aerEKgIoufVD_fodnuT)\n",
|
||||
"- [Getting Started with Fully Sharded Data Parallel(FSDP)](https://pytorch.org/tutorials/intermediate/FSDP_tutorial.html#:~:text=FSDP%20is%20a%20type%20of,sizes%20for%20our%20training%20job.)\n",
|
||||
"- [Fully Sharded Data Parallel: faster AI training with fewer GPUs](https://engineering.fb.com/2021/07/15/open-source/fsdp/)\n",
|
||||
"- [Hugging Face: dolly-v2-7b Model Card](https://huggingface.co/databricks/dolly-v2-7b)\n",
|
||||
"- [Hugging Face: Handling big models for inference](https://huggingface.co/docs/accelerate/usage_guides/big_modeling)"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.18"
|
||||
},
|
||||
"orphan": true
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,947 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Train a Pytorch Lightning Image Classifier\n",
|
||||
"\n",
|
||||
"<a id=\"try-anyscale-quickstart-lightning_mnist_example\" href=\"https://console.anyscale.com/register/ha?render_flow=ray&utm_source=ray_docs&utm_medium=docs&utm_campaign=lightning_mnist_example\">\n",
|
||||
" <img src=\"../../../_static/img/run-on-anyscale.svg\" alt=\"try-anyscale-quickstart\">\n",
|
||||
"</a>\n",
|
||||
"<br></br>\n",
|
||||
"\n",
|
||||
"This example introduces how to train a Pytorch Lightning Module using Ray Train {class}`TorchTrainer <ray.train.torch.TorchTrainer>`. It demonstrates how to train a basic neural network on the MNIST dataset with distributed data parallelism.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!pip install \"torchmetrics>=0.9\" \"lightning>=2.0\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"import numpy as np\n",
|
||||
"import random\n",
|
||||
"import torch\n",
|
||||
"import torch.nn as nn\n",
|
||||
"import torch.nn.functional as F\n",
|
||||
"from filelock import FileLock\n",
|
||||
"from torch.utils.data import DataLoader, random_split, Subset\n",
|
||||
"from torchmetrics import Accuracy\n",
|
||||
"from torchvision.datasets import MNIST\n",
|
||||
"from torchvision import transforms\n",
|
||||
"\n",
|
||||
"try:\n",
|
||||
" import lightning.pytorch as pl\n",
|
||||
" from lightning.pytorch.loggers import CSVLogger\n",
|
||||
"except ModuleNotFoundError:\n",
|
||||
" import pytorch_lightning as pl\n",
|
||||
" from pytorch_lightning.loggers import CSVLogger"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Prepare a dataset and module\n",
|
||||
"\n",
|
||||
"The Pytorch Lightning Trainer takes either `torch.utils.data.DataLoader` or `pl.LightningDataModule` as data inputs. You can continue using them without any changes with Ray Train. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class MNISTDataModule(pl.LightningDataModule):\n",
|
||||
" def __init__(self, batch_size=100):\n",
|
||||
" super().__init__()\n",
|
||||
" self.data_dir = os.getcwd()\n",
|
||||
" self.batch_size = batch_size\n",
|
||||
" self.transform = transforms.Compose(\n",
|
||||
" [transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))]\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" def setup(self, stage=None):\n",
|
||||
" with FileLock(f\"{self.data_dir}.lock\"):\n",
|
||||
" mnist = MNIST(\n",
|
||||
" self.data_dir, train=True, download=True, transform=self.transform\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" # Split data into train and val sets\n",
|
||||
" self.mnist_train, self.mnist_val = random_split(mnist, [55000, 5000])\n",
|
||||
"\n",
|
||||
" def train_dataloader(self):\n",
|
||||
" return DataLoader(self.mnist_train, batch_size=self.batch_size, num_workers=4)\n",
|
||||
"\n",
|
||||
" def val_dataloader(self):\n",
|
||||
" return DataLoader(self.mnist_val, batch_size=self.batch_size, num_workers=4)\n",
|
||||
"\n",
|
||||
" def test_dataloader(self):\n",
|
||||
" with FileLock(f\"{self.data_dir}.lock\"):\n",
|
||||
" self.mnist_test = MNIST(\n",
|
||||
" self.data_dir, train=False, download=True, transform=self.transform\n",
|
||||
" )\n",
|
||||
" return DataLoader(self.mnist_test, batch_size=self.batch_size, num_workers=4)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Next, define a simple multi-layer perception as the subclass of `pl.LightningModule`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class MNISTClassifier(pl.LightningModule):\n",
|
||||
" def __init__(self, lr=1e-3, feature_dim=128):\n",
|
||||
" torch.manual_seed(421)\n",
|
||||
" super(MNISTClassifier, self).__init__()\n",
|
||||
" self.save_hyperparameters()\n",
|
||||
"\n",
|
||||
" self.linear_relu_stack = nn.Sequential(\n",
|
||||
" nn.Linear(28 * 28, feature_dim),\n",
|
||||
" nn.ReLU(),\n",
|
||||
" nn.Linear(feature_dim, 10),\n",
|
||||
" nn.ReLU(),\n",
|
||||
" )\n",
|
||||
" self.lr = lr\n",
|
||||
" self.accuracy = Accuracy(task=\"multiclass\", num_classes=10, top_k=1)\n",
|
||||
" self.eval_loss = []\n",
|
||||
" self.eval_accuracy = []\n",
|
||||
" self.test_accuracy = []\n",
|
||||
" pl.seed_everything(888)\n",
|
||||
"\n",
|
||||
" def forward(self, x):\n",
|
||||
" x = x.view(-1, 28 * 28)\n",
|
||||
" x = self.linear_relu_stack(x)\n",
|
||||
" return x\n",
|
||||
"\n",
|
||||
" def training_step(self, batch, batch_idx):\n",
|
||||
" x, y = batch\n",
|
||||
" y_hat = self(x)\n",
|
||||
" loss = torch.nn.functional.cross_entropy(y_hat, y)\n",
|
||||
" self.log(\"train_loss\", loss)\n",
|
||||
" return loss\n",
|
||||
"\n",
|
||||
" def validation_step(self, val_batch, batch_idx):\n",
|
||||
" loss, acc = self._shared_eval(val_batch)\n",
|
||||
" self.log(\"val_accuracy\", acc)\n",
|
||||
" self.eval_loss.append(loss)\n",
|
||||
" self.eval_accuracy.append(acc)\n",
|
||||
" return {\"val_loss\": loss, \"val_accuracy\": acc}\n",
|
||||
"\n",
|
||||
" def test_step(self, test_batch, batch_idx):\n",
|
||||
" loss, acc = self._shared_eval(test_batch)\n",
|
||||
" self.test_accuracy.append(acc)\n",
|
||||
" self.log(\"test_accuracy\", acc, sync_dist=True, on_epoch=True)\n",
|
||||
" return {\"test_loss\": loss, \"test_accuracy\": acc}\n",
|
||||
"\n",
|
||||
" def _shared_eval(self, batch):\n",
|
||||
" x, y = batch\n",
|
||||
" logits = self.forward(x)\n",
|
||||
" loss = F.nll_loss(logits, y)\n",
|
||||
" acc = self.accuracy(logits, y)\n",
|
||||
" return loss, acc\n",
|
||||
"\n",
|
||||
" def on_validation_epoch_end(self):\n",
|
||||
" avg_loss = torch.stack(self.eval_loss).mean()\n",
|
||||
" avg_acc = torch.stack(self.eval_accuracy).mean()\n",
|
||||
" self.log(\"val_loss\", avg_loss, sync_dist=True)\n",
|
||||
" self.log(\"val_accuracy\", avg_acc, sync_dist=True)\n",
|
||||
" self.eval_loss.clear()\n",
|
||||
" self.eval_accuracy.clear()\n",
|
||||
" \n",
|
||||
" def configure_optimizers(self):\n",
|
||||
" optimizer = torch.optim.Adam(self.parameters(), lr=self.lr)\n",
|
||||
" return optimizer"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"You don't need to modify the definition of the PyTorch Lightning model or datamodule."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Define a training function\n",
|
||||
"\n",
|
||||
"This code defines a {ref}`training function <train-overview-training-function>` for each worker. Comparing the training function with the original PyTorch Lightning code, notice three main differences:\n",
|
||||
"\n",
|
||||
"- Distributed strategy: Use {class}`RayDDPStrategy <ray.train.lightning.RayDDPStrategy>`.\n",
|
||||
"- Cluster environment: Use {class}`RayLightningEnvironment <ray.train.lightning.RayLightningEnvironment>`.\n",
|
||||
"- Parallel devices: Always set to `devices=\"auto\"` to use all available devices configured by ``TorchTrainer``.\n",
|
||||
"\n",
|
||||
"See {ref}`Getting Started with PyTorch Lightning <train-pytorch-lightning>` for more information.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"For checkpoint reporting, Ray Train provides a minimal {class}`RayTrainReportCallback <ray.train.lightning.RayTrainReportCallback>` class that reports metrics and checkpoints at the end of each train epoch. For more complex checkpoint logic, implement custom callbacks. See {ref}`Saving and Loading Checkpoint <train-checkpointing>`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"use_gpu = True # Set to False if you want to run without GPUs\n",
|
||||
"num_workers = 4"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"try:\n",
|
||||
" import lightning.pytorch as pl\n",
|
||||
"except ModuleNotFoundError:\n",
|
||||
" import pytorch_lightning as pl\n",
|
||||
"\n",
|
||||
"from ray.train import RunConfig, ScalingConfig, CheckpointConfig\n",
|
||||
"from ray.train.torch import TorchTrainer\n",
|
||||
"from ray.train.lightning import (\n",
|
||||
" RayDDPStrategy,\n",
|
||||
" RayLightningEnvironment,\n",
|
||||
" RayTrainReportCallback,\n",
|
||||
" prepare_trainer,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"def train_func_per_worker():\n",
|
||||
" model = MNISTClassifier(lr=1e-3, feature_dim=128)\n",
|
||||
" datamodule = MNISTDataModule(batch_size=128)\n",
|
||||
"\n",
|
||||
" trainer = pl.Trainer(\n",
|
||||
" devices=\"auto\",\n",
|
||||
" strategy=RayDDPStrategy(),\n",
|
||||
" plugins=[RayLightningEnvironment()],\n",
|
||||
" callbacks=[RayTrainReportCallback()],\n",
|
||||
" max_epochs=10,\n",
|
||||
" accelerator=\"gpu\" if use_gpu else \"cpu\",\n",
|
||||
" log_every_n_steps=100,\n",
|
||||
" logger=CSVLogger(\"logs\"),\n",
|
||||
" )\n",
|
||||
" \n",
|
||||
" trainer = prepare_trainer(trainer)\n",
|
||||
" \n",
|
||||
" # Train model\n",
|
||||
" trainer.fit(model, datamodule=datamodule)\n",
|
||||
"\n",
|
||||
" # Evaluation on the test dataset\n",
|
||||
" trainer.test(model, datamodule=datamodule)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Now put everything together:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"scaling_config = ScalingConfig(num_workers=num_workers, use_gpu=use_gpu)\n",
|
||||
"\n",
|
||||
"run_config = RunConfig(\n",
|
||||
" name=\"ptl-mnist-example\",\n",
|
||||
" storage_path=\"/tmp/ray_results\",\n",
|
||||
" checkpoint_config=CheckpointConfig(\n",
|
||||
" num_to_keep=3,\n",
|
||||
" checkpoint_score_attribute=\"val_accuracy\",\n",
|
||||
" checkpoint_score_order=\"max\",\n",
|
||||
" ),\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"trainer = TorchTrainer(\n",
|
||||
" train_func_per_worker,\n",
|
||||
" scaling_config=scaling_config,\n",
|
||||
" run_config=run_config,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Now fit your trainer:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"<div class=\"tuneStatus\">\n",
|
||||
" <div style=\"display: flex;flex-direction: row\">\n",
|
||||
" <div style=\"display: flex;flex-direction: column;\">\n",
|
||||
" <h3>Tune Status</h3>\n",
|
||||
" <table>\n",
|
||||
"<tbody>\n",
|
||||
"<tr><td>Current time:</td><td>2023-08-07 23:41:11</td></tr>\n",
|
||||
"<tr><td>Running for: </td><td>00:00:39.80 </td></tr>\n",
|
||||
"<tr><td>Memory: </td><td>24.2/186.6 GiB </td></tr>\n",
|
||||
"</tbody>\n",
|
||||
"</table>\n",
|
||||
" </div>\n",
|
||||
" <div class=\"vDivider\"></div>\n",
|
||||
" <div class=\"systemInfo\">\n",
|
||||
" <h3>System Info</h3>\n",
|
||||
" Using FIFO scheduling algorithm.<br>Logical resource usage: 1.0/48 CPUs, 4.0/4 GPUs\n",
|
||||
" </div>\n",
|
||||
" \n",
|
||||
" </div>\n",
|
||||
" <div class=\"hDivider\"></div>\n",
|
||||
" <div class=\"trialStatus\">\n",
|
||||
" <h3>Trial Status</h3>\n",
|
||||
" <table>\n",
|
||||
"<thead>\n",
|
||||
"<tr><th>Trial name </th><th>status </th><th>loc </th><th style=\"text-align: right;\"> iter</th><th style=\"text-align: right;\"> total time (s)</th><th style=\"text-align: right;\"> train_loss</th><th style=\"text-align: right;\"> val_accuracy</th><th style=\"text-align: right;\"> val_loss</th></tr>\n",
|
||||
"</thead>\n",
|
||||
"<tbody>\n",
|
||||
"<tr><td>TorchTrainer_78346_00000</td><td>TERMINATED</td><td>10.0.6.244:120026</td><td style=\"text-align: right;\"> 10</td><td style=\"text-align: right;\"> 29.0221</td><td style=\"text-align: right;\"> 0.0315938</td><td style=\"text-align: right;\"> 0.970002</td><td style=\"text-align: right;\"> -12.3466</td></tr>\n",
|
||||
"</tbody>\n",
|
||||
"</table>\n",
|
||||
" </div>\n",
|
||||
"</div>\n",
|
||||
"<style>\n",
|
||||
".tuneStatus {\n",
|
||||
" color: var(--jp-ui-font-color1);\n",
|
||||
"}\n",
|
||||
".tuneStatus .systemInfo {\n",
|
||||
" display: flex;\n",
|
||||
" flex-direction: column;\n",
|
||||
"}\n",
|
||||
".tuneStatus td {\n",
|
||||
" white-space: nowrap;\n",
|
||||
"}\n",
|
||||
".tuneStatus .trialStatus {\n",
|
||||
" display: flex;\n",
|
||||
" flex-direction: column;\n",
|
||||
"}\n",
|
||||
".tuneStatus h3 {\n",
|
||||
" font-weight: bold;\n",
|
||||
"}\n",
|
||||
".tuneStatus .hDivider {\n",
|
||||
" border-bottom-width: var(--jp-border-width);\n",
|
||||
" border-bottom-color: var(--jp-border-color0);\n",
|
||||
" border-bottom-style: solid;\n",
|
||||
"}\n",
|
||||
".tuneStatus .vDivider {\n",
|
||||
" border-left-width: var(--jp-border-width);\n",
|
||||
" border-left-color: var(--jp-border-color0);\n",
|
||||
" border-left-style: solid;\n",
|
||||
" margin: 0.5em 1em 0.5em 1em;\n",
|
||||
"}\n",
|
||||
"</style>\n"
|
||||
],
|
||||
"text/plain": [
|
||||
"<IPython.core.display.HTML object>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"\u001b[2m\u001b[36m(TorchTrainer pid=120026)\u001b[0m Starting distributed worker processes: ['120176 (10.0.6.244)', '120177 (10.0.6.244)', '120178 (10.0.6.244)', '120179 (10.0.6.244)']\n",
|
||||
"\u001b[2m\u001b[36m(RayTrainWorker pid=120176)\u001b[0m Setting up process group for: env:// [rank=0, world_size=4]\n",
|
||||
"\u001b[2m\u001b[36m(RayTrainWorker pid=120176)\u001b[0m [rank: 0] Global seed set to 888\n",
|
||||
"\u001b[2m\u001b[36m(RayTrainWorker pid=120176)\u001b[0m GPU available: True (cuda), used: True\n",
|
||||
"\u001b[2m\u001b[36m(RayTrainWorker pid=120176)\u001b[0m TPU available: False, using: 0 TPU cores\n",
|
||||
"\u001b[2m\u001b[36m(RayTrainWorker pid=120176)\u001b[0m IPU available: False, using: 0 IPUs\n",
|
||||
"\u001b[2m\u001b[36m(RayTrainWorker pid=120176)\u001b[0m HPU available: False, using: 0 HPUs\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"\u001b[2m\u001b[36m(RayTrainWorker pid=120178)\u001b[0m Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz\n",
|
||||
"\u001b[2m\u001b[36m(RayTrainWorker pid=120178)\u001b[0m Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz to /tmp/ray_results/ptl-mnist-example/TorchTrainer_78346_00000_0_2023-08-07_23-40-31/rank_2/MNIST/raw/train-images-idx3-ubyte.gz\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
" 0%| | 0/9912422 [00:00<?, ?it/s]\n",
|
||||
"100%|██████████| 9912422/9912422 [00:00<00:00, 94562894.32it/s]\n",
|
||||
" 9%|▉ | 917504/9912422 [00:00<00:00, 9166590.91it/s]\n",
|
||||
"100%|██████████| 9912422/9912422 [00:00<00:00, 115619443.32it/s]\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"\u001b[2m\u001b[36m(RayTrainWorker pid=120179)\u001b[0m Extracting /tmp/ray_results/ptl-mnist-example/TorchTrainer_78346_00000_0_2023-08-07_23-40-31/rank_3/MNIST/raw/train-images-idx3-ubyte.gz to /tmp/ray_results/ptl-mnist-example/TorchTrainer_78346_00000_0_2023-08-07_23-40-31/rank_3/MNIST/raw\n",
|
||||
"\u001b[2m\u001b[36m(RayTrainWorker pid=120176)\u001b[0m \n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"\u001b[2m\u001b[36m(RayTrainWorker pid=120177)\u001b[0m Missing logger folder: logs/lightning_logs\n",
|
||||
"\u001b[2m\u001b[36m(RayTrainWorker pid=120176)\u001b[0m LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3]\n",
|
||||
"\u001b[2m\u001b[36m(RayTrainWorker pid=120176)\u001b[0m | Name | Type | Params\n",
|
||||
"\u001b[2m\u001b[36m(RayTrainWorker pid=120176)\u001b[0m ---------------------------------------------------------\n",
|
||||
"\u001b[2m\u001b[36m(RayTrainWorker pid=120176)\u001b[0m 0 | linear_relu_stack | Sequential | 101 K \n",
|
||||
"\u001b[2m\u001b[36m(RayTrainWorker pid=120176)\u001b[0m 1 | accuracy | MulticlassAccuracy | 0 \n",
|
||||
"\u001b[2m\u001b[36m(RayTrainWorker pid=120176)\u001b[0m ---------------------------------------------------------\n",
|
||||
"\u001b[2m\u001b[36m(RayTrainWorker pid=120176)\u001b[0m 101 K Trainable params\n",
|
||||
"\u001b[2m\u001b[36m(RayTrainWorker pid=120176)\u001b[0m 0 Non-trainable params\n",
|
||||
"\u001b[2m\u001b[36m(RayTrainWorker pid=120176)\u001b[0m 101 K Total params\n",
|
||||
"\u001b[2m\u001b[36m(RayTrainWorker pid=120176)\u001b[0m 0.407 Total estimated model params size (MB)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Sanity Checking: 0it [00:00, ?it/s])\u001b[0m \n",
|
||||
"Sanity Checking DataLoader 0: 0%| | 0/2 [00:00<?, ?it/s]\n",
|
||||
"Sanity Checking DataLoader 0: 100%|██████████| 2/2 [00:00<00:00, 2.69it/s]\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"\u001b[2m\u001b[36m(RayTrainWorker pid=120176)\u001b[0m /mnt/cluster_storage/pypi/lib/python3.9/site-packages/pytorch_lightning/trainer/connectors/logger_connector/result.py:432: PossibleUserWarning: It is recommended to use `self.log('val_accuracy', ..., sync_dist=True)` when logging on epoch level in distributed setting to accumulate the metric across devices.\n",
|
||||
"\u001b[2m\u001b[36m(RayTrainWorker pid=120176)\u001b[0m warning_cache.warn(\n",
|
||||
"\u001b[2m\u001b[36m(RayTrainWorker pid=120179)\u001b[0m [rank: 3] Global seed set to 888\u001b[32m [repeated 7x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/ray-logging.html#log-deduplication for more options.)\u001b[0m\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Epoch 0: 0%| | 0/108 [00:00<?, ?it/s] \n",
|
||||
"Epoch 0: 12%|█▏ | 13/108 [00:00<00:02, 39.35it/s, v_num=0]\n",
|
||||
"Epoch 0: 25%|██▌ | 27/108 [00:00<00:01, 59.26it/s, v_num=0]\n",
|
||||
"Epoch 0: 26%|██▌ | 28/108 [00:00<00:01, 61.03it/s, v_num=0]\n",
|
||||
"Epoch 0: 27%|██▋ | 29/108 [00:00<00:01, 62.76it/s, v_num=0]\n",
|
||||
"Epoch 0: 42%|████▏ | 45/108 [00:00<00:00, 81.02it/s, v_num=0]\n",
|
||||
"Epoch 0: 53%|█████▎ | 57/108 [00:00<00:00, 86.01it/s, v_num=0]\n",
|
||||
"Epoch 0: 64%|██████▍ | 69/108 [00:00<00:00, 88.63it/s, v_num=0]\n",
|
||||
"Epoch 0: 81%|████████ | 87/108 [00:00<00:00, 98.04it/s, v_num=0]\n",
|
||||
"Epoch 0: 81%|████████▏ | 88/108 [00:00<00:00, 98.69it/s, v_num=0]\n",
|
||||
"Epoch 0: 82%|████████▏ | 89/108 [00:00<00:00, 99.34it/s, v_num=0]\n",
|
||||
"Epoch 0: 96%|█████████▋| 104/108 [00:00<00:00, 104.14it/s, v_num=0]\n",
|
||||
"Epoch 0: 97%|█████████▋| 105/108 [00:01<00:00, 104.71it/s, v_num=0]\n",
|
||||
"Epoch 0: 98%|█████████▊| 106/108 [00:01<00:00, 105.22it/s, v_num=0]\n",
|
||||
"Epoch 0: 100%|██████████| 108/108 [00:01<00:00, 105.79it/s, v_num=0]\n",
|
||||
"Validation: 0it [00:00, ?it/s]\u001b[A76)\u001b[0m \n",
|
||||
"Validation: 0%| | 0/10 [00:00<?, ?it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 0%| | 0/10 [00:00<?, ?it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 10%|█ | 1/10 [00:00<00:00, 171.69it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 20%|██ | 2/10 [00:00<00:00, 200.99it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 30%|███ | 3/10 [00:00<00:00, 221.66it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 40%|████ | 4/10 [00:00<00:00, 215.50it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 50%|█████ | 5/10 [00:00<00:00, 194.14it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 60%|██████ | 6/10 [00:00<00:00, 205.63it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 70%|███████ | 7/10 [00:00<00:00, 215.27it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 80%|████████ | 8/10 [00:00<00:00, 216.26it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 90%|█████████ | 9/10 [00:00<00:00, 198.67it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 100%|██████████| 10/10 [00:00<00:00, 205.79it/s]\u001b[A\n",
|
||||
"Epoch 0: 100%|██████████| 108/108 [00:01<00:00, 79.84it/s, v_num=0] \n",
|
||||
" \u001b[A\n",
|
||||
"Epoch 1: 0%| | 0/108 [00:00<?, ?it/s, v_num=0] \n",
|
||||
"Epoch 1: 11%|█ | 12/108 [00:00<00:02, 32.36it/s, v_num=0]\n",
|
||||
"Epoch 1: 23%|██▎ | 25/108 [00:00<00:01, 50.16it/s, v_num=0]\n",
|
||||
"Epoch 1: 37%|███▋ | 40/108 [00:00<00:01, 65.95it/s, v_num=0]\n",
|
||||
"Epoch 1: 38%|███▊ | 41/108 [00:00<00:00, 67.05it/s, v_num=0]\n",
|
||||
"Epoch 1: 50%|█████ | 54/108 [00:00<00:00, 75.52it/s, v_num=0]\n",
|
||||
"Epoch 1: 51%|█████ | 55/108 [00:00<00:00, 76.40it/s, v_num=0]\n",
|
||||
"Epoch 1: 62%|██████▏ | 67/108 [00:00<00:00, 81.72it/s, v_num=0]\n",
|
||||
"\u001b[2m\u001b[36m(RayTrainWorker pid=120178)\u001b[0m Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz\u001b[32m [repeated 15x across cluster]\u001b[0m\n",
|
||||
"Epoch 1: 77%|███████▋ | 83/108 [00:00<00:00, 89.48it/s, v_num=0]\n",
|
||||
"\u001b[2m\u001b[36m(RayTrainWorker pid=120178)\u001b[0m Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz to /tmp/ray_results/ptl-mnist-example/TorchTrainer_78346_00000_0_2023-08-07_23-40-31/rank_2/MNIST/raw/t10k-labels-idx1-ubyte.gz\u001b[32m [repeated 15x across cluster]\u001b[0m\n",
|
||||
"Epoch 1: 78%|███████▊ | 84/108 [00:00<00:00, 89.21it/s, v_num=0]\n",
|
||||
"Epoch 1: 91%|█████████ | 98/108 [00:01<00:00, 93.27it/s, v_num=0]\n",
|
||||
"\u001b[2m\u001b[36m(RayTrainWorker pid=120178)\u001b[0m Extracting /tmp/ray_results/ptl-mnist-example/TorchTrainer_78346_00000_0_2023-08-07_23-40-31/rank_2/MNIST/raw/t10k-labels-idx1-ubyte.gz to /tmp/ray_results/ptl-mnist-example/TorchTrainer_78346_00000_0_2023-08-07_23-40-31/rank_2/MNIST/raw\u001b[32m [repeated 15x across cluster]\u001b[0m\n",
|
||||
"Epoch 1: 92%|█████████▏| 99/108 [00:01<00:00, 93.94it/s, v_num=0]\n",
|
||||
"Epoch 1: 93%|█████████▎| 100/108 [00:01<00:00, 94.57it/s, v_num=0]\n",
|
||||
"Epoch 1: 100%|██████████| 108/108 [00:01<00:00, 98.06it/s, v_num=0]\n",
|
||||
"Validation: 0it [00:00, ?it/s]\u001b[A76)\u001b[0m \n",
|
||||
"Validation: 0%| | 0/10 [00:00<?, ?it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 0%| | 0/10 [00:00<?, ?it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 10%|█ | 1/10 [00:00<00:00, 320.27it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 20%|██ | 2/10 [00:00<00:00, 291.99it/s]\u001b[A\n",
|
||||
"\u001b[2m\u001b[36m(RayTrainWorker pid=120176)\u001b[0m \u001b[32m [repeated 19x across cluster]\u001b[0m\n",
|
||||
"Validation DataLoader 0: 30%|███ | 3/10 [00:00<00:00, 291.61it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 40%|████ | 4/10 [00:00<00:00, 268.90it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 50%|█████ | 5/10 [00:00<00:00, 290.07it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 60%|██████ | 6/10 [00:00<00:00, 293.52it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 70%|███████ | 7/10 [00:00<00:00, 299.70it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 80%|████████ | 8/10 [00:00<00:00, 304.80it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 90%|█████████ | 9/10 [00:00<00:00, 310.16it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 100%|██████████| 10/10 [00:00<00:00, 303.63it/s]\u001b[A\n",
|
||||
"Epoch 1: 100%|██████████| 108/108 [00:01<00:00, 76.12it/s, v_num=0]\n",
|
||||
" \u001b[A\n",
|
||||
"Epoch 2: 0%| | 0/108 [00:00<?, ?it/s, v_num=0] \n",
|
||||
"Epoch 2: 7%|▋ | 8/108 [00:00<00:03, 25.23it/s, v_num=0]\n",
|
||||
"Epoch 2: 16%|█▌ | 17/108 [00:00<00:02, 39.73it/s, v_num=0]\n",
|
||||
"Epoch 2: 17%|█▋ | 18/108 [00:00<00:02, 41.60it/s, v_num=0]\n",
|
||||
"Epoch 2: 18%|█▊ | 19/108 [00:00<00:02, 43.49it/s, v_num=0]\n",
|
||||
"Epoch 2: 18%|█▊ | 19/108 [00:00<00:02, 43.46it/s, v_num=0]\n",
|
||||
"Epoch 2: 19%|█▊ | 20/108 [00:00<00:01, 45.27it/s, v_num=0]\n",
|
||||
"Epoch 2: 27%|██▋ | 29/108 [00:00<00:01, 53.08it/s, v_num=0]\n",
|
||||
"Epoch 2: 42%|████▏ | 45/108 [00:00<00:00, 69.12it/s, v_num=0]\n",
|
||||
"Epoch 2: 43%|████▎ | 46/108 [00:00<00:00, 70.31it/s, v_num=0]\n",
|
||||
"Epoch 2: 44%|████▎ | 47/108 [00:00<00:00, 71.51it/s, v_num=0]\n",
|
||||
"Epoch 2: 44%|████▍ | 48/108 [00:00<00:00, 72.71it/s, v_num=0]\n",
|
||||
"Epoch 2: 59%|█████▉ | 64/108 [00:00<00:00, 83.97it/s, v_num=0]\n",
|
||||
"Epoch 2: 75%|███████▌ | 81/108 [00:00<00:00, 93.77it/s, v_num=0]\n",
|
||||
"Epoch 2: 90%|████████▉ | 97/108 [00:00<00:00, 99.35it/s, v_num=0]\n",
|
||||
"Epoch 2: 100%|██████████| 108/108 [00:01<00:00, 101.71it/s, v_num=0]\n",
|
||||
"Validation: 0it [00:00, ?it/s]\u001b[A76)\u001b[0m \n",
|
||||
"Validation: 0%| | 0/10 [00:00<?, ?it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 0%| | 0/10 [00:00<?, ?it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 10%|█ | 1/10 [00:00<00:00, 212.13it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 20%|██ | 2/10 [00:00<00:00, 184.45it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 30%|███ | 3/10 [00:00<00:00, 228.42it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 40%|████ | 4/10 [00:00<00:00, 225.00it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 50%|█████ | 5/10 [00:00<00:00, 250.65it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 60%|██████ | 6/10 [00:00<00:00, 251.36it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 70%|███████ | 7/10 [00:00<00:00, 268.85it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 80%|████████ | 8/10 [00:00<00:00, 256.15it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 90%|█████████ | 9/10 [00:00<00:00, 269.87it/s]\u001b[A\n",
|
||||
"Epoch 2: 100%|██████████| 108/108 [00:01<00:00, 77.52it/s, v_num=0] it/s]\u001b[A\n",
|
||||
" \u001b[A\n",
|
||||
"Epoch 3: 0%| | 0/108 [00:00<?, ?it/s, v_num=0] \n",
|
||||
"Epoch 3: 8%|▊ | 9/108 [00:00<00:03, 25.68it/s, v_num=0]\n",
|
||||
"Epoch 3: 9%|▉ | 10/108 [00:00<00:03, 28.26it/s, v_num=0]\n",
|
||||
"Epoch 3: 20%|██ | 22/108 [00:00<00:01, 48.10it/s, v_num=0]\n",
|
||||
"Epoch 3: 21%|██▏ | 23/108 [00:00<00:01, 49.73it/s, v_num=0]\n",
|
||||
"Epoch 3: 22%|██▏ | 24/108 [00:00<00:01, 51.34it/s, v_num=0]\n",
|
||||
"Epoch 3: 23%|██▎ | 25/108 [00:00<00:01, 52.98it/s, v_num=0]\n",
|
||||
"Epoch 3: 37%|███▋ | 40/108 [00:00<00:00, 69.67it/s, v_num=0]\n",
|
||||
"Epoch 3: 51%|█████ | 55/108 [00:00<00:00, 80.93it/s, v_num=0]\n",
|
||||
"Epoch 3: 64%|██████▍ | 69/108 [00:00<00:00, 87.15it/s, v_num=0]\n",
|
||||
"Epoch 3: 65%|██████▍ | 70/108 [00:00<00:00, 88.04it/s, v_num=0]\n",
|
||||
"Epoch 3: 66%|██████▌ | 71/108 [00:00<00:00, 88.92it/s, v_num=0]\n",
|
||||
"Epoch 3: 77%|███████▋ | 83/108 [00:00<00:00, 92.62it/s, v_num=0]\n",
|
||||
"Epoch 3: 86%|████████▌ | 93/108 [00:01<00:00, 92.33it/s, v_num=0]\n",
|
||||
"Epoch 3: 87%|████████▋ | 94/108 [00:01<00:00, 92.93it/s, v_num=0]\n",
|
||||
"Epoch 3: 88%|████████▊ | 95/108 [00:01<00:00, 93.61it/s, v_num=0]\n",
|
||||
"Epoch 3: 100%|██████████| 108/108 [00:01<00:00, 97.43it/s, v_num=0]\n",
|
||||
"Validation: 0it [00:00, ?it/s]\u001b[A76)\u001b[0m \n",
|
||||
"Validation: 0%| | 0/10 [00:00<?, ?it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 0%| | 0/10 [00:00<?, ?it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 10%|█ | 1/10 [00:00<00:00, 308.50it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 20%|██ | 2/10 [00:00<00:00, 344.87it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 30%|███ | 3/10 [00:00<00:00, 375.98it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 40%|████ | 4/10 [00:00<00:00, 335.26it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 50%|█████ | 5/10 [00:00<00:00, 327.34it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 60%|██████ | 6/10 [00:00<00:00, 317.66it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 70%|███████ | 7/10 [00:00<00:00, 332.79it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 80%|████████ | 8/10 [00:00<00:00, 188.14it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 90%|█████████ | 9/10 [00:00<00:00, 201.21it/s]\u001b[A\n",
|
||||
"Epoch 3: 100%|██████████| 108/108 [00:01<00:00, 75.94it/s, v_num=0]6it/s]\u001b[A\n",
|
||||
" \u001b[A\n",
|
||||
"Epoch 4: 0%| | 0/108 [00:00<?, ?it/s, v_num=0] \n",
|
||||
"Epoch 4: 10%|█ | 11/108 [00:00<00:03, 30.09it/s, v_num=0]\n",
|
||||
"Epoch 4: 20%|██ | 22/108 [00:00<00:01, 46.96it/s, v_num=0]\n",
|
||||
"Epoch 4: 21%|██▏ | 23/108 [00:00<00:01, 47.88it/s, v_num=0]\n",
|
||||
"Epoch 4: 35%|███▌ | 38/108 [00:00<00:01, 65.26it/s, v_num=0]\n",
|
||||
"Epoch 4: 36%|███▌ | 39/108 [00:00<00:01, 65.73it/s, v_num=0]\n",
|
||||
"Epoch 4: 53%|█████▎ | 57/108 [00:00<00:00, 81.51it/s, v_num=0]\n",
|
||||
"Epoch 4: 54%|█████▎ | 58/108 [00:00<00:00, 82.56it/s, v_num=0]\n",
|
||||
"Epoch 4: 68%|██████▊ | 73/108 [00:00<00:00, 89.69it/s, v_num=0]\n",
|
||||
"Epoch 4: 69%|██████▊ | 74/108 [00:00<00:00, 90.53it/s, v_num=0]\n",
|
||||
"Epoch 4: 83%|████████▎ | 90/108 [00:00<00:00, 98.32it/s, v_num=0]\n",
|
||||
"Epoch 4: 98%|█████████▊| 106/108 [00:01<00:00, 103.12it/s, v_num=0]\n",
|
||||
"Epoch 4: 100%|██████████| 108/108 [00:01<00:00, 103.78it/s, v_num=0]\n",
|
||||
"Validation: 0it [00:00, ?it/s]\u001b[A76)\u001b[0m \n",
|
||||
"Validation: 0%| | 0/10 [00:00<?, ?it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 0%| | 0/10 [00:00<?, ?it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 10%|█ | 1/10 [00:00<00:00, 268.49it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 20%|██ | 2/10 [00:00<00:00, 298.62it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 30%|███ | 3/10 [00:00<00:00, 282.88it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 40%|████ | 4/10 [00:00<00:00, 256.50it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 50%|█████ | 5/10 [00:00<00:00, 276.28it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 60%|██████ | 6/10 [00:00<00:00, 268.05it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 70%|███████ | 7/10 [00:00<00:00, 276.18it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 80%|████████ | 8/10 [00:00<00:00, 290.08it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 90%|█████████ | 9/10 [00:00<00:00, 261.92it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 100%|██████████| 10/10 [00:00<00:00, 274.00it/s]\u001b[A\n",
|
||||
"Epoch 4: 100%|██████████| 108/108 [00:01<00:00, 78.54it/s, v_num=0] \n",
|
||||
" \u001b[A\n",
|
||||
"Epoch 5: 0%| | 0/108 [00:00<?, ?it/s, v_num=0] \n",
|
||||
"Epoch 5: 5%|▍ | 5/108 [00:00<00:06, 15.52it/s, v_num=0]\n",
|
||||
"\u001b[2m\u001b[36m(RayTrainWorker pid=120176)\u001b[0m \u001b[32m [repeated 9x across cluster]\u001b[0m\n",
|
||||
"Epoch 5: 17%|█▋ | 18/108 [00:00<00:02, 42.53it/s, v_num=0]\n",
|
||||
"Epoch 5: 26%|██▌ | 28/108 [00:00<00:01, 52.36it/s, v_num=0]\n",
|
||||
"Epoch 5: 27%|██▋ | 29/108 [00:00<00:01, 53.91it/s, v_num=0]\n",
|
||||
"Epoch 5: 28%|██▊ | 30/108 [00:00<00:01, 55.45it/s, v_num=0]\n",
|
||||
"Epoch 5: 29%|██▊ | 31/108 [00:00<00:01, 56.96it/s, v_num=0]\n",
|
||||
"Epoch 5: 37%|███▋ | 40/108 [00:00<00:01, 61.48it/s, v_num=0]\n",
|
||||
"Epoch 5: 38%|███▊ | 41/108 [00:00<00:01, 62.61it/s, v_num=0]\n",
|
||||
"Epoch 5: 39%|███▉ | 42/108 [00:00<00:01, 63.79it/s, v_num=0]\n",
|
||||
"Epoch 5: 40%|███▉ | 43/108 [00:00<00:01, 64.96it/s, v_num=0]\n",
|
||||
"Epoch 5: 48%|████▊ | 52/108 [00:00<00:00, 67.96it/s, v_num=0]\n",
|
||||
"Epoch 5: 49%|████▉ | 53/108 [00:00<00:00, 68.88it/s, v_num=0]\n",
|
||||
"Epoch 5: 50%|█████ | 54/108 [00:00<00:00, 69.77it/s, v_num=0]\n",
|
||||
"Epoch 5: 62%|██████▏ | 67/108 [00:00<00:00, 76.43it/s, v_num=0]\n",
|
||||
"Epoch 5: 78%|███████▊ | 84/108 [00:00<00:00, 85.56it/s, v_num=0]\n",
|
||||
"Epoch 5: 79%|███████▊ | 85/108 [00:00<00:00, 86.17it/s, v_num=0]\n",
|
||||
"Epoch 5: 93%|█████████▎| 100/108 [00:01<00:00, 92.27it/s, v_num=0]\n",
|
||||
"Epoch 5: 100%|██████████| 108/108 [00:01<00:00, 94.81it/s, v_num=0]\n",
|
||||
"Validation: 0it [00:00, ?it/s]\u001b[A76)\u001b[0m \n",
|
||||
"Validation: 0%| | 0/10 [00:00<?, ?it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 0%| | 0/10 [00:00<?, ?it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 10%|█ | 1/10 [00:00<00:00, 255.91it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 20%|██ | 2/10 [00:00<00:00, 206.50it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 30%|███ | 3/10 [00:00<00:00, 214.91it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 40%|████ | 4/10 [00:00<00:00, 236.35it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 50%|█████ | 5/10 [00:00<00:00, 240.05it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 60%|██████ | 6/10 [00:00<00:00, 234.90it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 70%|███████ | 7/10 [00:00<00:00, 240.78it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 80%|████████ | 8/10 [00:00<00:00, 252.00it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 90%|█████████ | 9/10 [00:00<00:00, 255.18it/s]\u001b[A\n",
|
||||
"Epoch 5: 100%|██████████| 108/108 [00:01<00:00, 72.36it/s, v_num=0]1it/s]\u001b[A\n",
|
||||
" \u001b[A\n",
|
||||
"Epoch 6: 0%| | 0/108 [00:00<?, ?it/s, v_num=0] \n",
|
||||
"Epoch 6: 15%|█▍ | 16/108 [00:00<00:02, 44.27it/s, v_num=0]\n",
|
||||
"Epoch 6: 25%|██▌ | 27/108 [00:00<00:01, 57.21it/s, v_num=0]\n",
|
||||
"Epoch 6: 38%|███▊ | 41/108 [00:00<00:00, 70.86it/s, v_num=0]\n",
|
||||
"Epoch 6: 39%|███▉ | 42/108 [00:00<00:00, 71.82it/s, v_num=0]\n",
|
||||
"Epoch 6: 55%|█████▍ | 59/108 [00:00<00:00, 85.97it/s, v_num=0]\n",
|
||||
"Epoch 6: 68%|██████▊ | 73/108 [00:00<00:00, 91.53it/s, v_num=0]\n",
|
||||
"Epoch 6: 81%|████████ | 87/108 [00:00<00:00, 96.88it/s, v_num=0]\n",
|
||||
"Epoch 6: 92%|█████████▏| 99/108 [00:00<00:00, 99.33it/s, v_num=0]\n",
|
||||
"Epoch 6: 93%|█████████▎| 100/108 [00:01<00:00, 98.66it/s, v_num=0]\n",
|
||||
"Epoch 6: 94%|█████████▎| 101/108 [00:01<00:00, 99.34it/s, v_num=0]\n",
|
||||
"Epoch 6: 100%|██████████| 108/108 [00:01<00:00, 102.79it/s, v_num=0]\n",
|
||||
"Validation: 0it [00:00, ?it/s]\u001b[A76)\u001b[0m \n",
|
||||
"Validation: 0%| | 0/10 [00:00<?, ?it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 0%| | 0/10 [00:00<?, ?it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 10%|█ | 1/10 [00:00<00:00, 197.51it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 20%|██ | 2/10 [00:00<00:00, 143.68it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 30%|███ | 3/10 [00:00<00:00, 156.17it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 40%|████ | 4/10 [00:00<00:00, 180.52it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 50%|█████ | 5/10 [00:00<00:00, 205.25it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 60%|██████ | 6/10 [00:00<00:00, 212.20it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 70%|███████ | 7/10 [00:00<00:00, 195.64it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 80%|████████ | 8/10 [00:00<00:00, 211.21it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 90%|█████████ | 9/10 [00:00<00:00, 225.13it/s]\u001b[A\n",
|
||||
"Epoch 6: 100%|██████████| 108/108 [00:01<00:00, 76.04it/s, v_num=0] it/s]\u001b[A\n",
|
||||
" \u001b[A\n",
|
||||
"Epoch 7: 0%| | 0/108 [00:00<?, ?it/s, v_num=0] \n",
|
||||
"Epoch 7: 11%|█ | 12/108 [00:00<00:02, 33.31it/s, v_num=0]\n",
|
||||
"Epoch 7: 20%|██ | 22/108 [00:00<00:01, 46.90it/s, v_num=0]\n",
|
||||
"Epoch 7: 22%|██▏ | 24/108 [00:00<00:01, 50.49it/s, v_num=0]\n",
|
||||
"Epoch 7: 31%|███▏ | 34/108 [00:00<00:01, 58.20it/s, v_num=0]\n",
|
||||
"Epoch 7: 32%|███▏ | 35/108 [00:00<00:01, 59.59it/s, v_num=0]\n",
|
||||
"Epoch 7: 33%|███▎ | 36/108 [00:00<00:01, 60.97it/s, v_num=0]\n",
|
||||
"Epoch 7: 48%|████▊ | 52/108 [00:00<00:00, 74.69it/s, v_num=0]\n",
|
||||
"Epoch 7: 64%|██████▍ | 69/108 [00:00<00:00, 85.96it/s, v_num=0]\n",
|
||||
"Epoch 7: 80%|███████▉ | 86/108 [00:00<00:00, 94.41it/s, v_num=0]\n",
|
||||
"Epoch 7: 81%|████████▏ | 88/108 [00:00<00:00, 95.91it/s, v_num=0]\n",
|
||||
"Epoch 7: 97%|█████████▋| 105/108 [00:01<00:00, 102.61it/s, v_num=0]\n",
|
||||
"Epoch 7: 100%|██████████| 108/108 [00:01<00:00, 103.00it/s, v_num=0]\n",
|
||||
"Validation: 0it [00:00, ?it/s]\u001b[A76)\u001b[0m \n",
|
||||
"Validation: 0%| | 0/10 [00:00<?, ?it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 0%| | 0/10 [00:00<?, ?it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 10%|█ | 1/10 [00:00<00:00, 215.46it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 20%|██ | 2/10 [00:00<00:00, 246.46it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 30%|███ | 3/10 [00:00<00:00, 264.39it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 40%|████ | 4/10 [00:00<00:00, 256.84it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 50%|█████ | 5/10 [00:00<00:00, 218.46it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 60%|██████ | 6/10 [00:00<00:00, 230.90it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 70%|███████ | 7/10 [00:00<00:00, 243.53it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 80%|████████ | 8/10 [00:00<00:00, 253.83it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 90%|█████████ | 9/10 [00:00<00:00, 249.22it/s]\u001b[A\n",
|
||||
"Epoch 7: 100%|██████████| 108/108 [00:01<00:00, 78.36it/s, v_num=0] it/s]\u001b[A\n",
|
||||
" \u001b[A\n",
|
||||
"Epoch 8: 0%| | 0/108 [00:00<?, ?it/s, v_num=0] \n",
|
||||
"Epoch 8: 7%|▋ | 8/108 [00:00<00:03, 25.72it/s, v_num=0]\n",
|
||||
"Epoch 8: 19%|█▊ | 20/108 [00:00<00:01, 47.54it/s, v_num=0]\n",
|
||||
"Epoch 8: 31%|███ | 33/108 [00:00<00:01, 62.61it/s, v_num=0]\n",
|
||||
"\u001b[2m\u001b[36m(RayTrainWorker pid=120176)\u001b[0m \u001b[32m [repeated 7x across cluster]\u001b[0m\n",
|
||||
"Epoch 8: 44%|████▎ | 47/108 [00:00<00:00, 74.45it/s, v_num=0]\n",
|
||||
"Epoch 8: 55%|█████▍ | 59/108 [00:00<00:00, 81.84it/s, v_num=0]\n",
|
||||
"Epoch 8: 56%|█████▌ | 60/108 [00:00<00:00, 80.82it/s, v_num=0]\n",
|
||||
"Epoch 8: 57%|█████▋ | 62/108 [00:00<00:00, 82.74it/s, v_num=0]\n",
|
||||
"Epoch 8: 58%|█████▊ | 63/108 [00:00<00:00, 83.69it/s, v_num=0]\n",
|
||||
"Epoch 8: 70%|███████ | 76/108 [00:00<00:00, 88.60it/s, v_num=0]\n",
|
||||
"Epoch 8: 85%|████████▌ | 92/108 [00:00<00:00, 96.53it/s, v_num=0]\n",
|
||||
"Epoch 8: 86%|████████▌ | 93/108 [00:00<00:00, 96.21it/s, v_num=0]\n",
|
||||
"Epoch 8: 87%|████████▋ | 94/108 [00:00<00:00, 96.72it/s, v_num=0]\n",
|
||||
"Epoch 8: 88%|████████▊ | 95/108 [00:00<00:00, 97.32it/s, v_num=0]\n",
|
||||
"Epoch 8: 89%|████████▉ | 96/108 [00:00<00:00, 98.03it/s, v_num=0]\n",
|
||||
"Epoch 8: 100%|██████████| 108/108 [00:01<00:00, 102.15it/s, v_num=0]\n",
|
||||
"Validation: 0it [00:00, ?it/s]\u001b[A76)\u001b[0m \n",
|
||||
"Validation: 0%| | 0/10 [00:00<?, ?it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 0%| | 0/10 [00:00<?, ?it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 10%|█ | 1/10 [00:00<00:00, 228.96it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 20%|██ | 2/10 [00:00<00:00, 220.63it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 30%|███ | 3/10 [00:00<00:00, 220.41it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 40%|████ | 4/10 [00:00<00:00, 208.74it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 50%|█████ | 5/10 [00:00<00:00, 221.74it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 60%|██████ | 6/10 [00:00<00:00, 243.64it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 70%|███████ | 7/10 [00:00<00:00, 253.60it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 80%|████████ | 8/10 [00:00<00:00, 254.93it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 90%|█████████ | 9/10 [00:00<00:00, 207.23it/s]\u001b[A\n",
|
||||
"Epoch 8: 100%|██████████| 108/108 [00:01<00:00, 78.28it/s, v_num=0] it/s]\u001b[A\n",
|
||||
" \u001b[A\n",
|
||||
"Epoch 9: 0%| | 0/108 [00:00<?, ?it/s, v_num=0] \n",
|
||||
"Epoch 9: 11%|█ | 12/108 [00:00<00:02, 33.03it/s, v_num=0]\n",
|
||||
"Epoch 9: 21%|██▏ | 23/108 [00:00<00:01, 48.82it/s, v_num=0]\n",
|
||||
"Epoch 9: 31%|███ | 33/108 [00:00<00:01, 58.62it/s, v_num=0]\n",
|
||||
"Epoch 9: 31%|███▏ | 34/108 [00:00<00:01, 58.61it/s, v_num=0]\n",
|
||||
"Epoch 9: 32%|███▏ | 35/108 [00:00<00:01, 59.89it/s, v_num=0]\n",
|
||||
"Epoch 9: 33%|███▎ | 36/108 [00:00<00:01, 61.11it/s, v_num=0]\n",
|
||||
"Epoch 9: 46%|████▋ | 50/108 [00:00<00:00, 71.95it/s, v_num=0]\n",
|
||||
"Epoch 9: 61%|██████ | 66/108 [00:00<00:00, 82.62it/s, v_num=0]\n",
|
||||
"Epoch 9: 70%|███████ | 76/108 [00:00<00:00, 83.77it/s, v_num=0]\n",
|
||||
"Epoch 9: 71%|███████▏ | 77/108 [00:00<00:00, 84.54it/s, v_num=0]\n",
|
||||
"Epoch 9: 72%|███████▏ | 78/108 [00:00<00:00, 85.33it/s, v_num=0]\n",
|
||||
"Epoch 9: 88%|████████▊ | 95/108 [00:01<00:00, 93.18it/s, v_num=0]\n",
|
||||
"Epoch 9: 100%|██████████| 108/108 [00:01<00:00, 98.27it/s, v_num=0]\n",
|
||||
"Validation: 0it [00:00, ?it/s]\u001b[A76)\u001b[0m \n",
|
||||
"Validation: 0%| | 0/10 [00:00<?, ?it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 0%| | 0/10 [00:00<?, ?it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 10%|█ | 1/10 [00:00<00:00, 305.42it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 20%|██ | 2/10 [00:00<00:00, 337.39it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 30%|███ | 3/10 [00:00<00:00, 368.65it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 40%|████ | 4/10 [00:00<00:00, 361.22it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 50%|█████ | 5/10 [00:00<00:00, 250.96it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 60%|██████ | 6/10 [00:00<00:00, 271.98it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 70%|███████ | 7/10 [00:00<00:00, 289.64it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 80%|████████ | 8/10 [00:00<00:00, 304.16it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 90%|█████████ | 9/10 [00:00<00:00, 184.87it/s]\u001b[A\n",
|
||||
"Validation DataLoader 0: 100%|██████████| 10/10 [00:00<00:00, 196.99it/s]\u001b[A\n",
|
||||
"Epoch 9: 100%|██████████| 108/108 [00:01<00:00, 74.63it/s, v_num=0]\n",
|
||||
" \u001b[A\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"\u001b[2m\u001b[36m(RayTrainWorker pid=120176)\u001b[0m `Trainer.fit` stopped: `max_epochs=10` reached.\n",
|
||||
"100%|██████████| 4542/4542 [00:00<00:00, 48474627.91it/s]\u001b[32m [repeated 14x across cluster]\u001b[0m\n",
|
||||
"100%|██████████| 9912422/9912422 [00:00<00:00, 90032420.31it/s]\u001b[32m [repeated 2x across cluster]\u001b[0m\n",
|
||||
"\u001b[2m\u001b[36m(RayTrainWorker pid=120176)\u001b[0m \u001b[32m [repeated 5x across cluster]\u001b[0m\n",
|
||||
"\u001b[2m\u001b[36m(RayTrainWorker pid=120178)\u001b[0m Missing logger folder: logs/lightning_logs\u001b[32m [repeated 2x across cluster]\u001b[0m\n",
|
||||
"\u001b[2m\u001b[36m(RayTrainWorker pid=120179)\u001b[0m LOCAL_RANK: 3 - CUDA_VISIBLE_DEVICES: [0,1,2,3]\u001b[32m [repeated 3x across cluster]\u001b[0m\n",
|
||||
"\u001b[2m\u001b[36m(RayTrainWorker pid=120176)\u001b[0m [rank: 0] Global seed set to 888\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Epoch 9: 100%|██████████| 108/108 [00:01<00:00, 66.61it/s, v_num=0]\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"\u001b[2m\u001b[36m(RayTrainWorker pid=120176)\u001b[0m /mnt/cluster_storage/pypi/lib/python3.9/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:225: PossibleUserWarning: Using `DistributedSampler` with the dataloaders. During `trainer.test()`, it is recommended to use `Trainer(devices=1, num_nodes=1)` to ensure each sample/batch gets evaluated exactly once. Otherwise, multi-device settings use `DistributedSampler` that replicates some samples to make sure all devices have same batch size in case of uneven inputs.\n",
|
||||
"\u001b[2m\u001b[36m(RayTrainWorker pid=120176)\u001b[0m rank_zero_warn(\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Testing DataLoader 0: 25%|██▌ | 5/20 [00:00<00:00, 146.57it/s]\n",
|
||||
"Testing DataLoader 0: 100%|██████████| 20/20 [00:00<00:00, 163.98it/s]\n",
|
||||
"Testing DataLoader 0: 100%|██████████| 20/20 [00:00<00:00, 125.34it/s]\n",
|
||||
"\u001b[2m\u001b[36m(RayTrainWorker pid=120176)\u001b[0m ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
|
||||
"\u001b[2m\u001b[36m(RayTrainWorker pid=120176)\u001b[0m ┃ Test metric ┃ DataLoader 0 ┃\n",
|
||||
"\u001b[2m\u001b[36m(RayTrainWorker pid=120176)\u001b[0m ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
|
||||
"\u001b[2m\u001b[36m(RayTrainWorker pid=120176)\u001b[0m │ test_accuracy │ 0.9740999937057495 │\n",
|
||||
"\u001b[2m\u001b[36m(RayTrainWorker pid=120176)\u001b[0m └───────────────────────────┴───────────────────────────┘\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"2023-08-07 23:41:11,072\tINFO tune.py:1145 -- Total run time: 39.92 seconds (39.80 seconds for the tuning loop).\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"result = trainer.fit()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Check training results and checkpoints"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"Result(\n",
|
||||
" metrics={'train_loss': 0.03159375861287117, 'val_accuracy': 0.9700015783309937, 'val_loss': -12.346583366394043, 'epoch': 9, 'step': 1080},\n",
|
||||
" path='/tmp/ray_results/ptl-mnist-example/TorchTrainer_78346_00000_0_2023-08-07_23-40-31',\n",
|
||||
" checkpoint=LegacyTorchCheckpoint(local_path=/tmp/ray_results/ptl-mnist-example/TorchTrainer_78346_00000_0_2023-08-07_23-40-31/checkpoint_000009)\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"result"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Validation Accuracy: 0.9700015783309937\n",
|
||||
"Trial Directory: /tmp/ray_results/ptl-mnist-example/TorchTrainer_78346_00000_0_2023-08-07_23-40-31\n",
|
||||
"['checkpoint_000007', 'checkpoint_000008', 'checkpoint_000009', 'events.out.tfevents.1691476838.ip-10-0-6-244', 'params.json', 'params.pkl', 'progress.csv', 'rank_0', 'rank_0.lock', 'rank_1', 'rank_1.lock', 'rank_2', 'rank_2.lock', 'rank_3', 'rank_3.lock', 'result.json']\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"print(\"Validation Accuracy: \", result.metrics[\"val_accuracy\"])\n",
|
||||
"print(\"Trial Directory: \", result.path)\n",
|
||||
"print(sorted(os.listdir(result.path)))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Ray Train saved three checkpoints(`checkpoint_000007`, `checkpoint_000008`, `checkpoint_000009`) in the trial directory. The following code retrieves the latest checkpoint from the fit results and loads it back into the model.\n",
|
||||
"\n",
|
||||
"If you lost the in-memory result object, you can restore the model from the checkpoint file. The checkpoint path is: `/tmp/ray_results/ptl-mnist-example/TorchTrainer_eb925_00000_0_2023-08-07_23-15-06/checkpoint_000009/checkpoint.ckpt`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Global seed set to 888\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"MNISTClassifier(\n",
|
||||
" (linear_relu_stack): Sequential(\n",
|
||||
" (0): Linear(in_features=784, out_features=128, bias=True)\n",
|
||||
" (1): ReLU()\n",
|
||||
" (2): Linear(in_features=128, out_features=10, bias=True)\n",
|
||||
" (3): ReLU()\n",
|
||||
" )\n",
|
||||
" (accuracy): MulticlassAccuracy()\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"checkpoint = result.checkpoint\n",
|
||||
"\n",
|
||||
"with checkpoint.as_directory() as ckpt_dir:\n",
|
||||
" best_model = MNISTClassifier.load_from_checkpoint(f\"{ckpt_dir}/checkpoint.ckpt\")\n",
|
||||
"\n",
|
||||
"best_model"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## See also\n",
|
||||
"\n",
|
||||
"* {ref}`Getting Started with PyTorch Lightning <train-pytorch-lightning>` for a tutorial on using Ray Train and PyTorch Lightning \n",
|
||||
"\n",
|
||||
"* {doc}`Ray Train Examples <../../examples>` for more use cases"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.11"
|
||||
},
|
||||
"orphan": true,
|
||||
"vscode": {
|
||||
"interpreter": {
|
||||
"hash": "a8c1140d108077f4faeb76b2438f85e4ed675f93d004359552883616a1acd54c"
|
||||
}
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user