run-llama--llama_index/docs/examples/observability/AimCallback.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "b4698442",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/run-llama/llama_index/blob/main/docs/examples/observability/AimCallback.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "fedcd46b",
   "metadata": {},
   "source": [
    "# Aim Callback\n",
    "\n",
    "Aim is an easy-to-use & supercharged open-source AI metadata tracker it logs all your AI metadata (experiments, prompts, etc) enables a UI to compare & observe them and SDK to query them programmatically. For more please see the [Github page](https://github.com/aimhubio/aim).\n",
    "\n",
    "In this demo, we show the capabilities of Aim for logging events while running queries within LlamaIndex. We use the AimCallback to store the outputs and showing how to explore them using Aim Text Explorer.\n",
    "\n",
    "\n",
    "**NOTE**: This is a beta feature. The usage within different classes and the API interface for the CallbackManager and AimCallback may change!"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "3e0c9e60",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "040821f4",
   "metadata": {},
   "source": [
    "If you're opening this Notebook on colab, you will probably need to install LlamaIndex 🦙."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d31a1e76",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install llama-index-callbacks-aim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4bd29997",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install llama-index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8e94187d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core.callbacks import CallbackManager\n",
    "from llama_index.callbacks.aim import AimCallback\n",
    "from llama_index.core import SummaryIndex\n",
    "from llama_index.core import SimpleDirectoryReader"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "17d1763e",
   "metadata": {},
   "source": [
    "Let's read the documents using `SimpleDirectoryReader` from 'examples/data/paul_graham'."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "8f6cca04",
   "metadata": {},
   "source": [
    "#### Download Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df144558",
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir -p 'data/paul_graham/'\n",
    "!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham/paul_graham_essay.txt'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "02e1e606",
   "metadata": {},
   "outputs": [],
   "source": [
    "docs = SimpleDirectoryReader(\"./data/paul_graham\").load_data()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "ee34d08b",
   "metadata": {},
   "source": [
    "Now lets initialize an AimCallback instance, and add it to the list of callback managers. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c667d70b",
   "metadata": {},
   "outputs": [],
   "source": [
    "aim_callback = AimCallback(repo=\"./\")\n",
    "callback_manager = CallbackManager([aim_callback])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "25851e27",
   "metadata": {},
   "source": [
    "In this snippet, we initialize a callback manager.\n",
    "Next, we create an instance of `SummaryIndex` class, by passing in the document reader and callback. After which we create a query engine which we will use to run queries on the index and retrieve relevant results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "32fac47b",
   "metadata": {},
   "outputs": [],
   "source": [
    "index = SummaryIndex.from_documents(docs, callback_manager=callback_manager)\n",
    "query_engine = index.as_query_engine()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "44f96768",
   "metadata": {},
   "source": [
    "Finally let's ask a question to the LM based on our provided document"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "11d4840b",
   "metadata": {},
   "outputs": [],
   "source": [
    "response = query_engine.query(\"What did the author do growing up?\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "4e69b186",
   "metadata": {},
   "source": [
    "The callback manager will log the `CBEventType.LLM` type of events as an Aim.Text, and we can explore the LM given prompt and the output in the Text Explorer. By first doing `aim up` and navigating by the given url."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}