wehub-resource-sync
161 lines
4.6 KiB
Plaintext
161 lines
4.6 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "a0b3171b",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Langsmith\n",
|
|
"## Dataset and Tracing Visualisation\n",
|
|
"\n",
|
|
"[Langsmith](https://docs.smith.langchain.com/) in a platform for building production-grade LLM applications from the langchain team. It helps you with tracing, debugging and evaluting LLM applications.\n",
|
|
"\n",
|
|
"The langsmith + ragas integrations offer 2 features\n",
|
|
"1. View the traces of ragas `evaluator` \n",
|
|
"2. Use ragas metrics in langchain evaluation - (soon)\n",
|
|
"\n",
|
|
"\n",
|
|
"## Tracing ragas metrics\n",
|
|
"\n",
|
|
"since ragas uses langchain under the hood all you have to do is setup langsmith and your traces will be logged.\n",
|
|
"\n",
|
|
"to setup langsmith make sure the following env-vars are set (you can read more in the [langsmith docs](https://docs.smith.langchain.com/#quick-start)\n",
|
|
"\n",
|
|
"```bash\n",
|
|
"export LANGCHAIN_TRACING_V2=true\n",
|
|
"export LANGCHAIN_ENDPOINT=https://api.smith.langchain.com\n",
|
|
"export LANGCHAIN_API_KEY=<your-api-key>\n",
|
|
"export LANGCHAIN_PROJECT=<your-project> # if not specified, defaults to \"default\"\n",
|
|
"```\n",
|
|
"\n",
|
|
"Once langsmith is setup, just run the evaluations as your normally would"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"id": "39375103",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Found cached dataset fiqa (/home/jjmachan/.cache/huggingface/datasets/vibrantlabsai___fiqa/ragas_eval/1.0.0/3dc7b639f5b4b16509a3299a2ceb78bf5fe98ee6b5fee25e7d5e4d290c88efb8)\n"
|
|
]
|
|
},
|
|
{
|
|
"data": {
|
|
"application/vnd.jupyter.widget-view+json": {
|
|
"model_id": "85ddc4fc4e184994892a8890792f06d8",
|
|
"version_major": 2,
|
|
"version_minor": 0
|
|
},
|
|
"text/plain": [
|
|
" 0%| | 0/1 [00:00<?, ?it/s]"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"evaluating with [context_precision]\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"100%|█████████████████████████████████████████████████████████████| 1/1 [00:23<00:00, 23.21s/it]\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"evaluating with [faithfulness]\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"100%|█████████████████████████████████████████████████████████████| 1/1 [00:36<00:00, 36.94s/it]\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"evaluating with [answer_relevancy]\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"100%|█████████████████████████████████████████████████████████████| 1/1 [00:10<00:00, 10.58s/it]\n"
|
|
]
|
|
},
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"{'context_precision': 0.5976, 'faithfulness': 0.8889, 'answer_relevancy': 0.9300}"
|
|
]
|
|
},
|
|
"execution_count": 1,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"from datasets import load_dataset\n",
|
|
"\n",
|
|
"from ragas import evaluate\n",
|
|
"from ragas.metrics import answer_relevancy, context_precision, faithfulness\n",
|
|
"\n",
|
|
"fiqa_eval = load_dataset(\"vibrantlabsai/fiqa\", \"ragas_eval\")\n",
|
|
"\n",
|
|
"result = evaluate(\n",
|
|
" fiqa_eval[\"baseline\"].select(range(3)),\n",
|
|
" metrics=[context_precision, faithfulness, answer_relevancy],\n",
|
|
")\n",
|
|
"\n",
|
|
"result"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "8ce1c649",
|
|
"metadata": {},
|
|
"source": [
|
|
"Voila! Now you can head over to your project and see the traces"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3 (ipykernel)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.10.12"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|