wehub-resource-sync
a0c8464e58
Build Package / build (ubuntu-latest) (push) Failing after 1s
CodeQL / Analyze (python) (push) Failing after 1s
Core Typecheck / core-typecheck (push) Failing after 1s
Linting / lint (push) Failing after 1s
llama-dev tests / test-llama-dev (push) Failing after 1s
Publish Sub-Package to PyPI if Needed / publish_subpackage_if_needed (push) Has been skipped
Sync Docs to Developer Hub / sync-docs (push) Failing after 0s
Build Package / build (windows-latest) (push) Has been cancelled
294 lines
8.6 KiB
Plaintext
294 lines
8.6 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"id": "d6509c3a",
|
|
"metadata": {},
|
|
"source": [
|
|
"<a href=\"https://colab.research.google.com/github/run-llama/llama_index/blob/main/docs/examples/observability/LangfuseCallbackHandler.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
|
|
"\n",
|
|
"# Langfuse Callback Handler\n",
|
|
"\n",
|
|
"⚠️ This integration is deprecated. We recommend using the new instrumentation-based integration with Langfuse as described [here](https://langfuse.com/docs/integrations/llama-index/get-started).\n",
|
|
"\n",
|
|
"This cookbook shows you how to use the Langfuse callback handler to monitor LlamaIndex applications.\n",
|
|
"\n",
|
|
"## What is Langfuse?\n",
|
|
"\n",
|
|
"[Langfuse](https://langfuse.com/docs) is an open source LLM engineering platform to help teams collaboratively debug, analyze and iterate on their LLM Applications. Langfuse offers a simple integration for automatic capture of [traces](https://langfuse.com/docs/tracing) and metrics generated in LlamaIndex applications. \n",
|
|
"\n",
|
|
"## How does it work?\n",
|
|
"\n",
|
|
"The `LangfuseCallbackHandler` is integrated with Langfuse and empowers you to seamlessly track and monitor performance, traces, and metrics of your LlamaIndex application. Detailed traces of the LlamaIndex context augmentation and the LLM querying processes are captured and can be inspected directly in the Langfuse UI."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "4a59a00e",
|
|
"metadata": {},
|
|
"source": [
|
|
""
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "3b9057da",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Setup"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "5d9dfc7f",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Install packages"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "49c3527e",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"%pip install llama-index llama-index-callbacks-langfuse"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "bc10630b",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Configure environment"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "4c256817",
|
|
"metadata": {},
|
|
"source": [
|
|
"If you haven't done yet, [sign up on Langfuse](https://cloud.langfuse.com/auth/sign-up) and obtain your API keys from the project settings."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "787e836d",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import os\n",
|
|
"\n",
|
|
"# Get keys for your project from the project settings page https://cloud.langfuse.com\n",
|
|
"os.environ[\"LANGFUSE_SECRET_KEY\"] = \"sk-lf-...\"\n",
|
|
"os.environ[\"LANGFUSE_PUBLIC_KEY\"] = \"pk-lf-...\"\n",
|
|
"os.environ[\"LANGFUSE_HOST\"] = \"https://cloud.langfuse.com\" # 🇪🇺 EU region\n",
|
|
"# os.environ[\"LANGFUSE_HOST\"] = \"https://us.cloud.langfuse.com\" # 🇺🇸 US region\n",
|
|
"\n",
|
|
"# OpenAI\n",
|
|
"os.environ[\"OPENAI_API_KEY\"] = \"sk-...\""
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "1fe2ba01",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Register the Langfuse callback handler"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "cfef9ddc",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### Option 1: Set global LlamaIndex handler"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "72afb2b9",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from llama_index.core import global_handler, set_global_handler\n",
|
|
"\n",
|
|
"set_global_handler(\"langfuse\")\n",
|
|
"langfuse_callback_handler = global_handler"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "0e6557d2",
|
|
"metadata": {},
|
|
"source": [
|
|
"#### Option 2: Use Langfuse callback directly"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "4bdd95bf",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from llama_index.core import Settings\n",
|
|
"from llama_index.core.callbacks import CallbackManager\n",
|
|
"from langfuse.llama_index import LlamaIndexCallbackHandler\n",
|
|
"\n",
|
|
"langfuse_callback_handler = LlamaIndexCallbackHandler()\n",
|
|
"Settings.callback_manager = CallbackManager([langfuse_callback_handler])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "e3e03ce7",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Flush events to Langfuse"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "e2c811ec",
|
|
"metadata": {},
|
|
"source": [
|
|
"The Langfuse SDKs queue and batches events in the background to reduce the number of network requests and improve overall performance. Before exiting your application, make sure all queued events have been flushed to Langfuse servers."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "4e28876c",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# ... your LlamaIndex calls here ...\n",
|
|
"\n",
|
|
"langfuse_callback_handler.flush()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "6b86f1b5",
|
|
"metadata": {},
|
|
"source": [
|
|
"Done!✨ Traces and metrics from your LlamaIndex application are now automatically tracked in Langfuse. If you construct a new index or query an LLM with your documents in context, your traces and metrics are immediately visible in the Langfuse UI. Next, let's take a look at how traces will look in Langfuse."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "1f0d4465",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Example"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "8a9f3428",
|
|
"metadata": {},
|
|
"source": [
|
|
"Fetch and save example data."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "aa303ae3",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"!mkdir -p 'data/'\n",
|
|
"!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham_essay.txt'"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "9f053996",
|
|
"metadata": {},
|
|
"source": [
|
|
"Run an example index construction, query, and chat."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "983cbedd",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from llama_index.core import SimpleDirectoryReader, VectorStoreIndex\n",
|
|
"\n",
|
|
"# Create index\n",
|
|
"documents = SimpleDirectoryReader(\"data\").load_data()\n",
|
|
"index = VectorStoreIndex.from_documents(documents)\n",
|
|
"\n",
|
|
"# Execute query\n",
|
|
"query_engine = index.as_query_engine()\n",
|
|
"query_response = query_engine.query(\"What did the author do growing up?\")\n",
|
|
"print(query_response)\n",
|
|
"\n",
|
|
"# Execute chat query\n",
|
|
"chat_engine = index.as_chat_engine()\n",
|
|
"chat_response = chat_engine.chat(\"What did the author do growing up?\")\n",
|
|
"print(chat_response)\n",
|
|
"\n",
|
|
"# As we want to immediately see result in Langfuse, we need to flush the callback handler\n",
|
|
"langfuse_callback_handler.flush()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "d5cdd88f",
|
|
"metadata": {},
|
|
"source": [
|
|
"Done!✨ You will now see traces of your index and query in your Langfuse project.\n",
|
|
"\n",
|
|
"Example traces (public links):\n",
|
|
"1. [Query](https://cloud.langfuse.com/project/cltipxbkn0000cdd7sbfbpovm/traces/f2e7f721-0940-4139-9b3a-e5cc9b0cb2d3)\n",
|
|
"2. [Query (chat)](https://cloud.langfuse.com/project/cltipxbkn0000cdd7sbfbpovm/traces/89c62a4d-e992-4923-a6b7-e2f27ae4cff3)\n",
|
|
"3. [Session](https://cloud.langfuse.com/project/cltipxbkn0000cdd7sbfbpovm/sessions/notebook-session-2)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "0b50845f",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 📚 More details\n",
|
|
"\n",
|
|
"Check out the full [Langfuse documentation](https://langfuse.com/docs) for more details on Langfuse's tracing and analytics capabilities and how to make most of this integration.\n",
|
|
"\n",
|
|
"## Feedback\n",
|
|
"\n",
|
|
"If you have any feedback or requests, please create a GitHub [Issue](https://github.com/orgs/langfuse/discussions) or share your idea with the community on [Discord](https://discord.langfuse.com/)."
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3 (ipykernel)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|