comet-ml--opik/apps/opik-documentation/documentation/docs/cookbook/langchain.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using Opik with Langchain\n",
    "\n",
    "For this guide, we will be performing a text to sql query generation task using LangChain. We will be using the Chinook database which contains the SQLite database of a music store with both employee, customer and invoice data.\n",
    "\n",
    "We will highlight three different parts of the workflow:\n",
    "\n",
    "1. Creating a synthetic dataset of questions\n",
    "2. Creating a LangChain chain to generate SQL queries\n",
    "3. Automating the evaluation of the SQL queries on the synthetic dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating an account on Comet.com\n",
    "\n",
    "[Comet](https://www.comet.com/site?from=llm&utm_source=opik&utm_medium=colab&utm_content=langchain&utm_campaign=opik) provides a hosted version of the Opik platform, [simply create an account](https://www.comet.com/signup?from=llm&utm_source=opik&utm_medium=colab&utm_content=langchain&utm_campaign=opik) and grab your API Key.\n",
    "\n",
    "> You can also run the Opik platform locally, see the [installation guide](https://www.comet.com/docs/opik/self-host/overview/?from=llm&utm_source=opik&utm_medium=colab&utm_content=langchain&utm_campaign=opik) for more information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install --upgrade --quiet opik langchain langchain-community langchain-openai"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import opik\n",
    "\n",
    "opik.configure(use_local=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparing our environment\n",
    "\n",
    "First, we will download the Chinook database and set up our different API keys."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import getpass\n",
    "\n",
    "if \"OPENAI_API_KEY\" not in os.environ:\n",
    "    os.environ[\"OPENAI_API_KEY\"] = getpass.getpass(\"Enter your OpenAI API key: \")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Download the relevant data\n",
    "import os\n",
    "from langchain_community.utilities import SQLDatabase\n",
    "\n",
    "import requests\n",
    "import os\n",
    "\n",
    "url = \"https://github.com/lerocha/chinook-database/raw/master/ChinookDatabase/DataSources/Chinook_Sqlite.sqlite\"\n",
    "filename = \"./data/chinook/Chinook_Sqlite.sqlite\"\n",
    "\n",
    "folder = os.path.dirname(filename)\n",
    "\n",
    "if not os.path.exists(folder):\n",
    "    os.makedirs(folder)\n",
    "\n",
    "if not os.path.exists(filename):\n",
    "    response = requests.get(url)\n",
    "    with open(filename, \"wb\") as file:\n",
    "        file.write(response.content)\n",
    "    print(\"Chinook database downloaded\")\n",
    "\n",
    "db = SQLDatabase.from_uri(f\"sqlite:///{filename}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating a synthetic dataset\n",
    "\n",
    "In order to create our synthetic dataset, we will be using the OpenAI API to generate 20 different questions that a user might ask based on the Chinook database.\n",
    "\n",
    "In order to ensure that the OpenAI API calls are being tracked, we will be using the `track_openai` function from the `opik` library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from opik.integrations.openai import track_openai\n",
    "from openai import OpenAI\n",
    "import json\n",
    "\n",
    "os.environ[\"OPIK_PROJECT_NAME\"] = \"langchain-integration-demo\"\n",
    "client = OpenAI()\n",
    "\n",
    "openai_client = track_openai(client)\n",
    "\n",
    "prompt = \"\"\"\n",
    "Create 20 different example questions a user might ask based on the Chinook Database.\n",
    "\n",
    "These questions should be complex and require the model to think. They should include complex joins and window functions to answer.\n",
    "\n",
    "Return the response as a json object with a \"result\" key and an array of strings with the question.\n",
    "\"\"\"\n",
    "\n",
    "completion = openai_client.chat.completions.create(\n",
    "    model=\"gpt-3.5-turbo\", messages=[{\"role\": \"user\", \"content\": prompt}]\n",
    ")\n",
    "\n",
    "print(completion.choices[0].message.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have our synthetic dataset, we can create a dataset in Comet and insert the questions into it.\n",
    "\n",
    "Since the insert methods in the SDK deduplicates items, we can insert 20 items and if the items already exist, Opik will automatically remove them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the synthetic dataset\n",
    "import opik\n",
    "\n",
    "synthetic_questions = json.loads(completion.choices[0].message.content)[\"result\"]\n",
    "\n",
    "client = opik.Opik()\n",
    "\n",
    "dataset = client.get_or_create_dataset(name=\"synthetic_questions\")\n",
    "dataset.insert([{\"question\": question} for question in synthetic_questions])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating a LangChain chain\n",
    "\n",
    "We will be using the `create_sql_query_chain` function from the `langchain` library to create a SQL query to answer the question.\n",
    "\n",
    "We will be using the `OpikTracer` class from the `opik` library to ensure that the LangChan trace are being tracked in Comet."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use langchain to create a SQL query to answer the question\n",
    "from langchain.chains import create_sql_query_chain\n",
    "from langchain_openai import ChatOpenAI\n",
    "from opik.integrations.langchain import OpikTracer\n",
    "\n",
    "opik_tracer = OpikTracer(tags=[\"simple_chain\"])\n",
    "\n",
    "llm = ChatOpenAI(model=\"gpt-3.5-turbo\", temperature=0)\n",
    "chain = create_sql_query_chain(llm, db).with_config({\"callbacks\": [opik_tracer]})\n",
    "response = chain.invoke({\"question\": \"How many employees are there ?\"})\n",
    "response\n",
    "\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Automating the evaluation\n",
    "\n",
    "In order to ensure our LLM application is working correctly, we will test it on our synthetic dataset.\n",
    "\n",
    "For this we will be using the `evaluate` function from the `opik` library. We will evaluate the application using a custom metric that checks if the SQL query is valid."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from opik import Opik, track\n",
    "from opik.evaluation import evaluate\n",
    "from opik.evaluation.metrics import base_metric, score_result\n",
    "from typing import Any\n",
    "\n",
    "\n",
    "class ValidSQLQuery(base_metric.BaseMetric):\n",
    "    def __init__(self, name: str, db: Any):\n",
    "        self.name = name\n",
    "        self.db = db\n",
    "\n",
    "    def score(self, output: str, **ignored_kwargs: Any):\n",
    "        # Add you logic here\n",
    "\n",
    "        try:\n",
    "            db.run(output)\n",
    "            return score_result.ScoreResult(\n",
    "                name=self.name, value=1, reason=\"Query ran successfully\"\n",
    "            )\n",
    "        except Exception as e:\n",
    "            return score_result.ScoreResult(name=self.name, value=0, reason=str(e))\n",
    "\n",
    "\n",
    "valid_sql_query = ValidSQLQuery(name=\"valid_sql_query\", db=db)\n",
    "\n",
    "client = Opik()\n",
    "dataset = client.get_dataset(\"synthetic_questions\")\n",
    "\n",
    "\n",
    "@track()\n",
    "def llm_chain(input: str) -> str:\n",
    "    response = chain.invoke({\"question\": input})\n",
    "\n",
    "    return response\n",
    "\n",
    "\n",
    "def evaluation_task(item):\n",
    "    response = llm_chain(item[\"question\"])\n",
    "\n",
    "    return {\"output\": response}\n",
    "\n",
    "\n",
    "res = evaluate(\n",
    "    experiment_name=\"SQL question answering\",\n",
    "    dataset=dataset,\n",
    "    task=evaluation_task,\n",
    "    scoring_metrics=[valid_sql_query],\n",
    "    nb_samples=20,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The evaluation results are now uploaded to the Opik platform and can be viewed in the UI.\n",
    "\n",
    "![LangChain Evaluation](https://raw.githubusercontent.com/comet-ml/opik/main/apps/opik-documentation/documentation/fern/img/cookbook/langchain_cookbook.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "py312_opik",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}