wehub-resource-sync
178 lines
6.2 KiB
Plaintext
178 lines
6.2 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"id": "3b05af3b",
|
|
"metadata": {},
|
|
"source": [
|
|
"(tune-comet-ref)=\n",
|
|
"\n",
|
|
"# Using Comet with Tune\n",
|
|
"\n",
|
|
"<a id=\"try-anyscale-quickstart-tune-comet\" href=\"https://console.anyscale.com/register/ha?render_flow=ray&utm_source=ray_docs&utm_medium=docs&utm_campaign=tune-comet\">\n",
|
|
" <img src=\"../../_static/img/run-on-anyscale.svg\" alt=\"try-anyscale-quickstart\">\n",
|
|
"</a>\n",
|
|
"<br></br>\n",
|
|
"\n",
|
|
"[Comet](https://www.comet.ml/site/) is a tool to manage and optimize the\n",
|
|
"entire ML lifecycle, from experiment tracking, model optimization and dataset\n",
|
|
"versioning to model production monitoring.\n",
|
|
"\n",
|
|
"```{image} /images/comet_logo_full.png\n",
|
|
":align: center\n",
|
|
":alt: Comet\n",
|
|
":height: 120px\n",
|
|
":target: https://www.comet.ml/site/\n",
|
|
"```\n",
|
|
"\n",
|
|
"```{contents}\n",
|
|
":backlinks: none\n",
|
|
":local: true\n",
|
|
"```\n",
|
|
"\n",
|
|
"## Example\n",
|
|
"\n",
|
|
"To illustrate logging your trial results to Comet, we'll define a simple training function\n",
|
|
"that simulates a `loss` metric:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"id": "19e3c389",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import numpy as np\n",
|
|
"from ray import tune\n",
|
|
"\n",
|
|
"\n",
|
|
"def train_function(config):\n",
|
|
" for i in range(30):\n",
|
|
" loss = config[\"mean\"] + config[\"sd\"] * np.random.randn()\n",
|
|
" tune.report({\"loss\": loss})"
|
|
]
|
|
},
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"id": "6fb69a24",
|
|
"metadata": {},
|
|
"source": [
|
|
"Now, given that you provide your Comet API key and your project name like so:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"id": "993d5be6",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"api_key = \"YOUR_COMET_API_KEY\"\n",
|
|
"project_name = \"YOUR_COMET_PROJECT_NAME\""
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"id": "e9ce0d76",
|
|
"metadata": {
|
|
"tags": [
|
|
"remove-cell"
|
|
]
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# This cell is hidden from the rendered notebook. It makes the \n",
|
|
"from unittest.mock import MagicMock\n",
|
|
"from ray.air.integrations.comet import CometLoggerCallback\n",
|
|
"\n",
|
|
"CometLoggerCallback._logger_process_cls = MagicMock\n",
|
|
"api_key = \"abc\"\n",
|
|
"project_name = \"test\""
|
|
]
|
|
},
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"id": "d792a1b0",
|
|
"metadata": {},
|
|
"source": [
|
|
"You can add a Comet logger by specifying the `callbacks` argument in your `RunConfig()` accordingly:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "dbb761e7",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from ray.air.integrations.comet import CometLoggerCallback\n",
|
|
"\n",
|
|
"tuner = tune.Tuner(\n",
|
|
" train_function,\n",
|
|
" tune_config=tune.TuneConfig(\n",
|
|
" metric=\"loss\",\n",
|
|
" mode=\"min\",\n",
|
|
" ),\n",
|
|
" run_config=tune.RunConfig(\n",
|
|
" callbacks=[\n",
|
|
" CometLoggerCallback(\n",
|
|
" api_key=api_key, project_name=project_name, tags=[\"comet_example\"]\n",
|
|
" )\n",
|
|
" ],\n",
|
|
" ),\n",
|
|
" param_space={\"mean\": tune.grid_search([1, 2, 3]), \"sd\": tune.uniform(0.2, 0.8)},\n",
|
|
")\n",
|
|
"results = tuner.fit()\n",
|
|
"\n",
|
|
"print(results.get_best_result().config)"
|
|
]
|
|
},
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"id": "d7e46189",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Tune Comet Logger\n",
|
|
"\n",
|
|
"Ray Tune offers an integration with Comet through the `CometLoggerCallback`,\n",
|
|
"which automatically logs metrics and parameters reported to Tune to the Comet UI.\n",
|
|
"\n",
|
|
"Click on the following dropdown to see this callback API in detail:\n",
|
|
"\n",
|
|
"```{eval-rst}\n",
|
|
".. autoclass:: ray.air.integrations.comet.CometLoggerCallback\n",
|
|
" :noindex:\n",
|
|
"```"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3 (ipykernel)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.7.7"
|
|
},
|
|
"orphan": true
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|