wehub/ray-project--ray
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity
Files
main
ray-project--ray/doc/source/tune/examples/tune-aim.ipynb
T
wehub-resource-sync f1825c8ceb chore: import upstream snapshot with attribution
2026-07-13 13:17:40 +08:00

413 lines
23 KiB
Plaintext
Raw Permalink Blame History

{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"id": "ecad719c",
"metadata": {},
"source": [
"(tune-aim-ref)=\n",
"\n",
"# Using Aim with Tune\n",
"\n",
"<a id=\"try-anyscale-quickstart-tune-aim\" href=\"https://console.anyscale.com/register/ha?render_flow=ray&utm_source=ray_docs&utm_medium=docs&utm_campaign=tune-aim\">\n",
" <img src=\"../../_static/img/run-on-anyscale.svg\" alt=\"try-anyscale-quickstart\">\n",
"</a>\n",
"<br></br>\n",
"\n",
"[Aim](https://aimstack.io) is an easy-to-use and supercharged open-source experiment tracker.\n",
"Aim logs your training runs, enables a well-designed UI to compare them, and provides an API to query them programmatically.\n",
"\n",
"```{image} /images/aim_logo_full.png\n",
":align: center\n",
":alt: Aim\n",
":width: 100%\n",
":target: https://aimstack.io\n",
"```\n",
"\n",
"Ray Tune currently offers built-in integration with Aim.\n",
"The {ref}`AimLoggerCallback <tune-aim-logger>` automatically logs metrics that are reported to Tune by using the Aim API.\n",
"\n",
"\n",
"```{contents}\n",
":backlinks: none\n",
":local: true\n",
"```\n",
"\n",
"## Logging Tune Hyperparameter Configurations and Results to Aim\n",
"\n",
"The following example demonstrates how the `AimLoggerCallback` can be used in a Tune experiment.\n",
"Begin by installing and importing the necessary modules:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1290b5b5",
"metadata": {},
"outputs": [],
"source": [
"%pip install aim\n",
"%pip install ray[tune]"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "100bcf8a",
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"\n",
"import ray\n",
"from ray import tune\n",
"from ray.tune.logger.aim import AimLoggerCallback"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "9346c0f6",
"metadata": {},
"source": [
"Next, define a simple `train_function`, which is a [`Trainable`](trainable-docs) that reports a loss to Tune.\n",
"The objective function itself is not important for this example, as our main focus is on the integration with Aim."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "e8b4fc4d",
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"def train_function(config):\n",
" for _ in range(50):\n",
" loss = config[\"mean\"] + config[\"sd\"] * np.random.randn()\n",
" tune.report({\"loss\": loss})"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "831eed42",
"metadata": {},
"source": [
"Here is an example of how you can use the `AimLoggerCallback` with simple grid-search Tune experiment.\n",
"The logger will log each of the 9 grid-search trials as separate Aim runs."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "52988599",
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-02-07 00:04:11,228\tINFO worker.py:1544 -- Started a local Ray instance. View the dashboard at \u001b[1m\u001b[32mhttp://127.0.0.1:8265 \u001b[39m\u001b[22m\n"
]
},
{
"data": {
"text/html": [
"<div class=\"tuneStatus\">\n",
" <div style=\"display: flex;flex-direction: row\">\n",
" <div style=\"display: flex;flex-direction: column;\">\n",
" <h3>Tune Status</h3>\n",
" <table>\n",
"<tbody>\n",
"<tr><td>Current time:</td><td>2023-02-07 00:04:19</td></tr>\n",
"<tr><td>Running for: </td><td>00:00:06.86 </td></tr>\n",
"<tr><td>Memory: </td><td>32.8/64.0 GiB </td></tr>\n",
"</tbody>\n",
"</table>\n",
" </div>\n",
" <div class=\"vDivider\"></div>\n",
" <div class=\"systemInfo\">\n",
" <h3>System Info</h3>\n",
" Using FIFO scheduling algorithm.<br>Resources requested: 0/10 CPUs, 0/0 GPUs, 0.0/26.93 GiB heap, 0.0/2.0 GiB objects\n",
" </div>\n",
" \n",
" </div>\n",
" <div class=\"hDivider\"></div>\n",
" <div class=\"trialStatus\">\n",
" <h3>Trial Status</h3>\n",
" <table>\n",
"<thead>\n",
"<tr><th>Trial name </th><th>status </th><th>loc </th><th style=\"text-align: right;\"> mean</th><th style=\"text-align: right;\"> sd</th><th style=\"text-align: right;\"> iter</th><th style=\"text-align: right;\"> total time (s)</th><th style=\"text-align: right;\"> loss</th></tr>\n",
"</thead>\n",
"<tbody>\n",
"<tr><td>train_function_01a3b_00000</td><td>TERMINATED</td><td>127.0.0.1:10277</td><td style=\"text-align: right;\"> 1</td><td style=\"text-align: right;\">0.385428</td><td style=\"text-align: right;\"> 50</td><td style=\"text-align: right;\"> 4.48031</td><td style=\"text-align: right;\">1.01928</td></tr>\n",
"<tr><td>train_function_01a3b_00001</td><td>TERMINATED</td><td>127.0.0.1:10296</td><td style=\"text-align: right;\"> 2</td><td style=\"text-align: right;\">0.819716</td><td style=\"text-align: right;\"> 50</td><td style=\"text-align: right;\"> 2.97272</td><td style=\"text-align: right;\">3.01491</td></tr>\n",
"<tr><td>train_function_01a3b_00002</td><td>TERMINATED</td><td>127.0.0.1:10301</td><td style=\"text-align: right;\"> 3</td><td style=\"text-align: right;\">0.769197</td><td style=\"text-align: right;\"> 50</td><td style=\"text-align: right;\"> 2.39572</td><td style=\"text-align: right;\">3.87155</td></tr>\n",
"<tr><td>train_function_01a3b_00003</td><td>TERMINATED</td><td>127.0.0.1:10307</td><td style=\"text-align: right;\"> 4</td><td style=\"text-align: right;\">0.29466 </td><td style=\"text-align: right;\"> 50</td><td style=\"text-align: right;\"> 2.41568</td><td style=\"text-align: right;\">4.1507 </td></tr>\n",
"<tr><td>train_function_01a3b_00004</td><td>TERMINATED</td><td>127.0.0.1:10313</td><td style=\"text-align: right;\"> 5</td><td style=\"text-align: right;\">0.152208</td><td style=\"text-align: right;\"> 50</td><td style=\"text-align: right;\"> 1.68383</td><td style=\"text-align: right;\">5.10225</td></tr>\n",
"<tr><td>train_function_01a3b_00005</td><td>TERMINATED</td><td>127.0.0.1:10321</td><td style=\"text-align: right;\"> 6</td><td style=\"text-align: right;\">0.879814</td><td style=\"text-align: right;\"> 50</td><td style=\"text-align: right;\"> 1.54015</td><td style=\"text-align: right;\">6.20238</td></tr>\n",
"<tr><td>train_function_01a3b_00006</td><td>TERMINATED</td><td>127.0.0.1:10329</td><td style=\"text-align: right;\"> 7</td><td style=\"text-align: right;\">0.487499</td><td style=\"text-align: right;\"> 50</td><td style=\"text-align: right;\"> 1.44706</td><td style=\"text-align: right;\">7.79551</td></tr>\n",
"<tr><td>train_function_01a3b_00007</td><td>TERMINATED</td><td>127.0.0.1:10333</td><td style=\"text-align: right;\"> 8</td><td style=\"text-align: right;\">0.639783</td><td style=\"text-align: right;\"> 50</td><td style=\"text-align: right;\"> 1.4261 </td><td style=\"text-align: right;\">7.94189</td></tr>\n",
"<tr><td>train_function_01a3b_00008</td><td>TERMINATED</td><td>127.0.0.1:10341</td><td style=\"text-align: right;\"> 9</td><td style=\"text-align: right;\">0.12285 </td><td style=\"text-align: right;\"> 50</td><td style=\"text-align: right;\"> 1.07701</td><td style=\"text-align: right;\">8.82304</td></tr>\n",
"</tbody>\n",
"</table>\n",
" </div>\n",
"</div>\n",
"<style>\n",
".tuneStatus {\n",
" color: var(--jp-ui-font-color1);\n",
"}\n",
".tuneStatus .systemInfo {\n",
" display: flex;\n",
" flex-direction: column;\n",
"}\n",
".tuneStatus td {\n",
" white-space: nowrap;\n",
"}\n",
".tuneStatus .trialStatus {\n",
" display: flex;\n",
" flex-direction: column;\n",
"}\n",
".tuneStatus h3 {\n",
" font-weight: bold;\n",
"}\n",
".tuneStatus .hDivider {\n",
" border-bottom-width: var(--jp-border-width);\n",
" border-bottom-color: var(--jp-border-color0);\n",
" border-bottom-style: solid;\n",
"}\n",
".tuneStatus .vDivider {\n",
" border-left-width: var(--jp-border-width);\n",
" border-left-color: var(--jp-border-color0);\n",
" border-left-style: solid;\n",
" margin: 0.5em 1em 0.5em 1em;\n",
"}\n",
"</style>\n"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<div class=\"trialProgress\">\n",
" <h3>Trial Progress</h3>\n",
" <table>\n",
"<thead>\n",
"<tr><th>Trial name </th><th>date </th><th>done </th><th>episodes_total </th><th>experiment_id </th><th>experiment_tag </th><th>hostname </th><th style=\"text-align: right;\"> iterations_since_restore</th><th style=\"text-align: right;\"> loss</th><th>node_ip </th><th style=\"text-align: right;\"> pid</th><th style=\"text-align: right;\"> time_since_restore</th><th style=\"text-align: right;\"> time_this_iter_s</th><th style=\"text-align: right;\"> time_total_s</th><th style=\"text-align: right;\"> timestamp</th><th style=\"text-align: right;\"> timesteps_since_restore</th><th>timesteps_total </th><th style=\"text-align: right;\"> training_iteration</th><th>trial_id </th><th style=\"text-align: right;\"> warmup_time</th></tr>\n",
"</thead>\n",
"<tbody>\n",
"<tr><td>train_function_01a3b_00000</td><td>2023-02-07_00-04-18</td><td>True </td><td> </td><td>c8447fdceea6436c9edd6f030a5b1d82</td><td>0_mean=1,sd=0.3854</td><td>Justins-MacBook-Pro-16</td><td style=\"text-align: right;\"> 50</td><td style=\"text-align: right;\">1.01928</td><td>127.0.0.1</td><td style=\"text-align: right;\">10277</td><td style=\"text-align: right;\"> 4.48031</td><td style=\"text-align: right;\"> 0.013865 </td><td style=\"text-align: right;\"> 4.48031</td><td style=\"text-align: right;\"> 1675757058</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 50</td><td>01a3b_00000</td><td style=\"text-align: right;\"> 0.00264072</td></tr>\n",
"<tr><td>train_function_01a3b_00001</td><td>2023-02-07_00-04-18</td><td>True </td><td> </td><td>7dd6d3ee24244a0885b354c285064728</td><td>1_mean=2,sd=0.8197</td><td>Justins-MacBook-Pro-16</td><td style=\"text-align: right;\"> 50</td><td style=\"text-align: right;\">3.01491</td><td>127.0.0.1</td><td style=\"text-align: right;\">10296</td><td style=\"text-align: right;\"> 2.97272</td><td style=\"text-align: right;\"> 0.0584073 </td><td style=\"text-align: right;\"> 2.97272</td><td style=\"text-align: right;\"> 1675757058</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 50</td><td>01a3b_00001</td><td style=\"text-align: right;\"> 0.0316792 </td></tr>\n",
"<tr><td>train_function_01a3b_00002</td><td>2023-02-07_00-04-18</td><td>True </td><td> </td><td>e3da49ebad034c4b8fdaf0aa87927b1a</td><td>2_mean=3,sd=0.7692</td><td>Justins-MacBook-Pro-16</td><td style=\"text-align: right;\"> 50</td><td style=\"text-align: right;\">3.87155</td><td>127.0.0.1</td><td style=\"text-align: right;\">10301</td><td style=\"text-align: right;\"> 2.39572</td><td style=\"text-align: right;\"> 0.0695491 </td><td style=\"text-align: right;\"> 2.39572</td><td style=\"text-align: right;\"> 1675757058</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 50</td><td>01a3b_00002</td><td style=\"text-align: right;\"> 0.0315411 </td></tr>\n",
"<tr><td>train_function_01a3b_00003</td><td>2023-02-07_00-04-18</td><td>True </td><td> </td><td>95c60c4f67c4481ebccff25b0a49e75d</td><td>3_mean=4,sd=0.2947</td><td>Justins-MacBook-Pro-16</td><td style=\"text-align: right;\"> 50</td><td style=\"text-align: right;\">4.1507 </td><td>127.0.0.1</td><td style=\"text-align: right;\">10307</td><td style=\"text-align: right;\"> 2.41568</td><td style=\"text-align: right;\"> 0.0175381 </td><td style=\"text-align: right;\"> 2.41568</td><td style=\"text-align: right;\"> 1675757058</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 50</td><td>01a3b_00003</td><td style=\"text-align: right;\"> 0.0310779 </td></tr>\n",
"<tr><td>train_function_01a3b_00004</td><td>2023-02-07_00-04-18</td><td>True </td><td> </td><td>a216253cb41e47caa229e65488deb019</td><td>4_mean=5,sd=0.1522</td><td>Justins-MacBook-Pro-16</td><td style=\"text-align: right;\"> 50</td><td style=\"text-align: right;\">5.10225</td><td>127.0.0.1</td><td style=\"text-align: right;\">10313</td><td style=\"text-align: right;\"> 1.68383</td><td style=\"text-align: right;\"> 0.064441 </td><td style=\"text-align: right;\"> 1.68383</td><td style=\"text-align: right;\"> 1675757058</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 50</td><td>01a3b_00004</td><td style=\"text-align: right;\"> 0.00450182</td></tr>\n",
"<tr><td>train_function_01a3b_00005</td><td>2023-02-07_00-04-18</td><td>True </td><td> </td><td>23834104277f476cb99d9c696281fceb</td><td>5_mean=6,sd=0.8798</td><td>Justins-MacBook-Pro-16</td><td style=\"text-align: right;\"> 50</td><td style=\"text-align: right;\">6.20238</td><td>127.0.0.1</td><td style=\"text-align: right;\">10321</td><td style=\"text-align: right;\"> 1.54015</td><td style=\"text-align: right;\"> 0.00910306</td><td style=\"text-align: right;\"> 1.54015</td><td style=\"text-align: right;\"> 1675757058</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 50</td><td>01a3b_00005</td><td style=\"text-align: right;\"> 0.0480251 </td></tr>\n",
"<tr><td>train_function_01a3b_00006</td><td>2023-02-07_00-04-18</td><td>True </td><td> </td><td>15f650121df747c3bd2720481d47b265</td><td>6_mean=7,sd=0.4875</td><td>Justins-MacBook-Pro-16</td><td style=\"text-align: right;\"> 50</td><td style=\"text-align: right;\">7.79551</td><td>127.0.0.1</td><td style=\"text-align: right;\">10329</td><td style=\"text-align: right;\"> 1.44706</td><td style=\"text-align: right;\"> 0.00600386</td><td style=\"text-align: right;\"> 1.44706</td><td style=\"text-align: right;\"> 1675757058</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 50</td><td>01a3b_00006</td><td style=\"text-align: right;\"> 0.00202489</td></tr>\n",
"<tr><td>train_function_01a3b_00007</td><td>2023-02-07_00-04-19</td><td>True </td><td> </td><td>78b1673cf2034ed99135b80a0cb31e0e</td><td>7_mean=8,sd=0.6398</td><td>Justins-MacBook-Pro-16</td><td style=\"text-align: right;\"> 50</td><td style=\"text-align: right;\">7.94189</td><td>127.0.0.1</td><td style=\"text-align: right;\">10333</td><td style=\"text-align: right;\"> 1.4261 </td><td style=\"text-align: right;\"> 0.00225306</td><td style=\"text-align: right;\"> 1.4261 </td><td style=\"text-align: right;\"> 1675757059</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 50</td><td>01a3b_00007</td><td style=\"text-align: right;\"> 0.00209713</td></tr>\n",
"<tr><td>train_function_01a3b_00008</td><td>2023-02-07_00-04-19</td><td>True </td><td> </td><td>c7f5d86154cb46b6aa27bef523edcd6f</td><td>8_mean=9,sd=0.1228</td><td>Justins-MacBook-Pro-16</td><td style=\"text-align: right;\"> 50</td><td style=\"text-align: right;\">8.82304</td><td>127.0.0.1</td><td style=\"text-align: right;\">10341</td><td style=\"text-align: right;\"> 1.07701</td><td style=\"text-align: right;\"> 0.00291467</td><td style=\"text-align: right;\"> 1.07701</td><td style=\"text-align: right;\"> 1675757059</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 50</td><td>01a3b_00008</td><td style=\"text-align: right;\"> 0.00240111</td></tr>\n",
"</tbody>\n",
"</table>\n",
"</div>\n",
"<style>\n",
".trialProgress {\n",
" display: flex;\n",
" flex-direction: column;\n",
" color: var(--jp-ui-font-color1);\n",
"}\n",
".trialProgress h3 {\n",
" font-weight: bold;\n",
"}\n",
".trialProgress td {\n",
" white-space: nowrap;\n",
"}\n",
"</style>\n"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-02-07 00:04:19,366\tINFO tune.py:798 -- Total run time: 7.38 seconds (6.85 seconds for the tuning loop).\n"
]
},
{
"data": {
"text/plain": [
"<ray.tune.result_grid.ResultGrid at 0x137de07c0>"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"tuner = tune.Tuner(\n",
" train_function,\n",
" run_config=tune.RunConfig(\n",
" callbacks=[AimLoggerCallback()],\n",
" storage_path=\"/tmp/ray_results\",\n",
" name=\"aim_example\",\n",
" ),\n",
" param_space={\n",
" \"mean\": tune.grid_search([1, 2, 3, 4, 5, 6, 7, 8, 9]),\n",
" \"sd\": tune.uniform(0.1, 0.9),\n",
" },\n",
" tune_config=tune.TuneConfig(\n",
" metric=\"loss\",\n",
" mode=\"min\",\n",
" ),\n",
")\n",
"tuner.fit()\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "941f25f2",
"metadata": {},
"source": [
"When the script executes, a grid-search is carried out and the results are saved to the Aim repo,\n",
"stored at the default location -- the experiment log directory (in this case, it's at `/tmp/ray_results/aim_example`).\n",
"\n",
"### More Configuration Options for Aim\n",
"\n",
"In the example above, we used the default configuration for the `AimLoggerCallback`.\n",
"There are a few options that can be configured as arguments to the callback. For example,\n",
"setting `AimLoggerCallback(repo=\"/path/to/repo\")` will log results to the Aim repo at that\n",
"filepath, which could be useful if you have a central location where the results of multiple\n",
"Tune experiments are stored. Relative paths to the working directory where Tune script is\n",
"launched can be used as well. By default, the repo will be set to the experiment log\n",
"directory. See [the API reference](tune-aim-logger) for more configurations.\n",
"\n",
"## Launching the Aim UI\n",
"\n",
"Now that we have logged our results to the Aim repository, we can view it in Aim's web UI.\n",
"To do this, we first find the directory where the Aim repository lives, then we use\n",
"the Aim CLI to launch the web interface."
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "880f55aa",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"--------------------------------------------------------------------------\n",
" Aim UI collects anonymous usage analytics. \n",
" Read how to opt-out here: \n",
" https://aimstack.readthedocs.io/en/latest/community/telemetry.html \n",
"--------------------------------------------------------------------------\n",
"\u001b[33mRunning Aim UI on repo `<Repo#-5734997863388805469 path=/tmp/ray_results/aim_example/.aim read_only=None>`\u001b[0m\n",
"Open http://127.0.0.1:43800\n",
"Press Ctrl+C to exit\n",
"^C\n"
]
}
],
"source": [
"# Uncomment the following line to launch the Aim UI!\n",
"#!aim up --repo=/tmp/ray_results/aim_example"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "adbe661a",
"metadata": {},
"source": [
"After launching the Aim UI, we can open the web interface at `localhost:43800`."
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "7bb97157",
"metadata": {},
"source": [
"```{image} /images/aim_example_metrics_page.png\n",
":align: center\n",
":alt: Aim Metrics Explorer\n",
":target: https://aimstack.readthedocs.io/en/latest/ui/pages/explorers.html#metrics-explorer\n",
"```"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "2f6e9138",
"metadata": {},
"source": [
"The next sections contain more in-depth information on the API of the Tune-Aim integration.\n",
"\n",
"## Tune Aim Logger API\n",
"\n",
"(tune-aim-logger)=\n",
"\n",
"```{eval-rst}\n",
".. autoclass:: ray.tune.logger.aim.AimLoggerCallback\n",
" :noindex:\n",
"```"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "0ebd1904",
"metadata": {},
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "ray_dev_py38",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.13"
},
"orphan": true,
"vscode": {
"interpreter": {
"hash": "265d195fda5292fe8f69c6e37c435a5634a1ed3b6799724e66a975f68fa21517"
}
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Reference in New Issue View Git Blame Copy Permalink