wehub/flagopen--flagembedding
1
0
Fork 0
Code Issues Pull Requests Actions 2 Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 13:39:21 +08:00
commit bc34f6df14
1149 changed files with 328099 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/source/tutorial/2_Metrics/2.1.ipynb
+798
View File
@@ -0,0 +1,798 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "0d0f87e9-657d-46b9-a3f0-ebc1bf0656bd",
"metadata": {},
"source": [
"# Similarity"
]
},
{
"cell_type": "markdown",
"id": "00c817d5",
"metadata": {},
"source": [
"In this section, we will introduce several different ways to measure similarity."
]
},
{
"cell_type": "markdown",
"id": "dae49384-2450-425c-b050-c27d3c07d8e7",
"metadata": {
"tags": []
},
"source": [
"## 1. Jaccard Similarity"
]
},
{
"cell_type": "markdown",
"id": "03266267-2d6d-4124-9702-f61e0510586c",
"metadata": {},
"source": [
"Before directly calculate the similarity between embedding vectors, let's first take a look at the primal method for measuring how similar two sentenses are: Jaccard similarity.\n",
"\n",
"**Definition:** For sets $A$ and $B$, the Jaccard index, or the Jaccard similarity coefficient between them is the size of their intersection divided by the size of their union:\n",
"$$J(A,B)=\\frac{|A\\cap B|}{|A\\cup B|}$$\n",
"\n",
"The value of $J(A,B)$ falls in the range of $[0, 1]$."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "bed533e1-a17c-4595-bdff-7f4a29e4deb3",
"metadata": {
"execution": {
"iopub.execute_input": "2024-07-17T03:12:47.091346Z",
"iopub.status.busy": "2024-07-17T03:12:47.091019Z",
"iopub.status.idle": "2024-07-17T03:12:47.094401Z",
"shell.execute_reply": "2024-07-17T03:12:47.093967Z",
"shell.execute_reply.started": "2024-07-17T03:12:47.091327Z"
},
"tags": []
},
"outputs": [],
"source": [
"def jaccard_similarity(sentence1, sentence2):\n",
" set1 = set(sentence1.split(\" \"))\n",
" set2 = set(sentence2.split(\" \"))\n",
" intersection = set1.intersection(set2)\n",
" union = set1.union(set2)\n",
" return len(intersection)/len(union)"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "ea766de8-572d-4eca-91f7-284a121e8edb",
"metadata": {
"ExecutionIndicator": {
"show": true
},
"execution": {
"iopub.execute_input": "2024-07-17T03:14:06.133012Z",
"iopub.status.busy": "2024-07-17T03:14:06.132502Z",
"iopub.status.idle": "2024-07-17T03:14:06.135483Z",
"shell.execute_reply": "2024-07-17T03:14:06.135044Z",
"shell.execute_reply.started": "2024-07-17T03:14:06.132992Z"
},
"tags": []
},
"outputs": [],
"source": [
"s1 = \"Hawaii is a wonderful place for holiday\"\n",
"s2 = \"Peter's favorite place to spend his holiday is Hawaii\"\n",
"s3 = \"Anna enjoys baking during her holiday\""
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "b359ff4e-21a1-489a-ad46-ba53e974dc48",
"metadata": {
"execution": {
"iopub.execute_input": "2024-07-17T03:13:34.646320Z",
"iopub.status.busy": "2024-07-17T03:13:34.645942Z",
"iopub.status.idle": "2024-07-17T03:13:34.649389Z",
"shell.execute_reply": "2024-07-17T03:13:34.648998Z",
"shell.execute_reply.started": "2024-07-17T03:13:34.646302Z"
},
"tags": []
},
"outputs": [
{
"data": {
"text/plain": [
"0.3333333333333333"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"jaccard_similarity(s1, s2)"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "069868a9-d379-4d55-8a23-835a2972d079",
"metadata": {
"execution": {
"iopub.execute_input": "2024-07-17T03:14:13.727400Z",
"iopub.status.busy": "2024-07-17T03:14:13.726949Z",
"iopub.status.idle": "2024-07-17T03:14:13.730545Z",
"shell.execute_reply": "2024-07-17T03:14:13.730121Z",
"shell.execute_reply.started": "2024-07-17T03:14:13.727381Z"
},
"tags": []
},
"outputs": [
{
"data": {
"text/plain": [
"0.08333333333333333"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"jaccard_similarity(s1, s3)"
]
},
{
"cell_type": "markdown",
"id": "b0323128",
"metadata": {},
"source": [
"We can see that sentence 1 and 2 are sharing 'Hawaii', 'place', and 'holiday'. Thus getting a larger score of similarity (0.333) than that (0.083) of the sentence 1 and 3 that only share 'holiday'."
]
},
{
"cell_type": "markdown",
"id": "b509fa6c-87ac-4c59-b40e-fda95fd036d9",
"metadata": {
"tags": []
},
"source": [
"## 2. Euclidean Distance"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "9da366b8-427f-4e8f-b3e6-b453050f0591",
"metadata": {
"ExecutionIndicator": {
"show": true
},
"execution": {
"iopub.execute_input": "2024-07-17T02:30:37.643857Z",
"iopub.status.busy": "2024-07-17T02:30:37.643302Z",
"iopub.status.idle": "2024-07-17T02:30:37.647921Z",
"shell.execute_reply": "2024-07-17T02:30:37.647513Z",
"shell.execute_reply.started": "2024-07-17T02:30:37.643840Z"
},
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"tensor([[5., 2., 2., 6.]]) tensor([[4., 6., 6., 4.]])\n"
]
}
],
"source": [
"import torch\n",
"\n",
"A = torch.randint(1, 7, (1, 4), dtype=torch.float32)\n",
"B = torch.randint(1, 7, (1, 4), dtype=torch.float32)\n",
"print(A, B)"
]
},
{
"cell_type": "markdown",
"id": "6c068bb3-90ce-4266-8335-e3fb2ad3e996",
"metadata": {},
"source": [
"**Definition:** For vectors $A$ and $B$, the Euclidean distance or L2 distance between them is defined as:\n",
"$$d(A, B) = \\|A-B\\|_2 = \\sqrt{\\sum_{i=1}^n (A_i-B_i)^2}$$\n",
"\n",
"The value of $d(A, B)$ falls in the range of [0, $+\\infty$). Since this is the measurement of distance, the closer the value is to 0, the more similar the two vector is. And the larger the value is, the two vectors are more dissimilar."
]
},
{
"cell_type": "markdown",
"id": "1d6c734d-cc03-4dd1-bb9e-3243006dcff4",
"metadata": {},
"source": [
"You can calculate Euclidean distance step by step or directly call *torch.cdist()*"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "0773acf4-eb53-4058-85da-af82af20c469",
"metadata": {
"ExecutionIndicator": {
"show": true
},
"execution": {
"iopub.execute_input": "2024-07-17T02:32:45.240684Z",
"iopub.status.busy": "2024-07-17T02:32:45.240216Z",
"iopub.status.idle": "2024-07-17T02:32:45.244248Z",
"shell.execute_reply": "2024-07-17T02:32:45.243843Z",
"shell.execute_reply.started": "2024-07-17T02:32:45.240665Z"
},
"tags": []
},
"outputs": [
{
"data": {
"text/plain": [
"6.082762718200684"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"dist = torch.sqrt(torch.sum(torch.pow(torch.subtract(A, B), 2), dim=-1))\n",
"dist.item()"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "1dd45446-f7d6-4aab-b078-1d34f0a949e4",
"metadata": {
"ExecutionIndicator": {
"show": true
},
"execution": {
"iopub.execute_input": "2024-07-17T02:32:57.551560Z",
"iopub.status.busy": "2024-07-17T02:32:57.550896Z",
"iopub.status.idle": "2024-07-17T02:32:57.555031Z",
"shell.execute_reply": "2024-07-17T02:32:57.554638Z",
"shell.execute_reply.started": "2024-07-17T02:32:57.551536Z"
},
"tags": []
},
"outputs": [
{
"data": {
"text/plain": [
"6.082762718200684"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"torch.cdist(A, B, p=2).item()"
]
},
{
"cell_type": "markdown",
"id": "da4435c0-98da-4397-8a45-c954dd3ada56",
"metadata": {},
"source": [
"### (Maximum inner-product search)"
]
},
{
"cell_type": "markdown",
"id": "0e0fa5c2-e619-4a0f-a785-9cc209f1503b",
"metadata": {
"tags": []
},
"source": [
"## 3. Cosine Similarity"
]
},
{
"cell_type": "markdown",
"id": "790e1ce3-1468-4819-a956-fc8eac690d89",
"metadata": {},
"source": [
"For vectors $A$ and $B$, their cosine similarity is defined as:\n",
"$$\\cos(\\theta)=\\frac{A\\cdot B}{\\|A\\|\\|B\\|}$$\n",
"\n",
"The value of $\\cos(\\theta)$ falls in the range of $[-1, 1]$. Different from Euclidean distance, close to -1 denotes not similar at all and close to +1 means very similar."
]
},
{
"cell_type": "markdown",
"id": "d0a64b4b-5caf-4bee-be0f-2e26b1c7ed6e",
"metadata": {
"tags": []
},
"source": [
"### 3.1 Naive Approach"
]
},
{
"cell_type": "markdown",
"id": "350cc48d-6e73-4e20-86dd-c05d1238ef60",
"metadata": {},
"source": [
"The naive approach is just expanding the expression:\n",
"$$\\frac{A\\cdot B}{\\|A\\|\\|B\\|}=\\frac{\\sum_{i=1}^{i=n}A_i B_i}{\\sqrt{\\sum_{i=1}^{n}A_i^2}\\cdot\\sqrt{\\sum_{i=1}^{n}B_i^2}}$$"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "20c7cff0-55a7-4222-9e5a-f5450171fb00",
"metadata": {
"ExecutionIndicator": {
"show": true
},
"execution": {
"iopub.execute_input": "2024-07-17T02:24:35.239550Z",
"iopub.status.busy": "2024-07-17T02:24:35.239073Z",
"iopub.status.idle": "2024-07-17T02:24:35.242844Z",
"shell.execute_reply": "2024-07-17T02:24:35.242417Z",
"shell.execute_reply.started": "2024-07-17T02:24:35.239531Z"
},
"tags": []
},
"outputs": [],
"source": [
"# Compute the dot product of A and B\n",
"dot_prod = sum(a*b for a, b in zip(A[0], B[0]))\n",
"\n",
"# Compute the magnitude of A and B\n",
"A_norm = torch.sqrt(sum(a*a for a in A[0]))\n",
"B_norm = torch.sqrt(sum(b*b for b in B[0]))"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "f4dce1fb-9cff-4a0d-bc7f-a503be6a37ae",
"metadata": {
"ExecutionIndicator": {
"show": true
},
"execution": {
"iopub.execute_input": "2024-07-17T02:24:36.533667Z",
"iopub.status.busy": "2024-07-17T02:24:36.533224Z",
"iopub.status.idle": "2024-07-17T02:24:36.536611Z",
"shell.execute_reply": "2024-07-17T02:24:36.536181Z",
"shell.execute_reply.started": "2024-07-17T02:24:36.533650Z"
},
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0.802726686000824\n"
]
}
],
"source": [
"cos_1 = dot_prod / (A_norm * B_norm)\n",
"print(cos_1.item())"
]
},
{
"cell_type": "markdown",
"id": "4665f38f-c1f1-42dd-914d-d1d69c038e88",
"metadata": {
"tags": []
},
"source": [
"### 3.2 PyTorch Implementation"
]
},
{
"cell_type": "markdown",
"id": "6154391d-1dea-4673-8502-b496cf87d4b0",
"metadata": {},
"source": [
"The naive approach has few issues:\n",
"- There are chances of losing precision in the numerator and the denominator\n",
"- Losing precision may cause the computed cosine similarity > 1.0\n",
"\n",
"Thus PyTorch uses the following way:\n",
"\n",
"$$\n",
"\\frac{A\\cdot B}{\\|A\\|\\|B\\|}=\\frac{A}{\\|A\\|}\\cdot\\frac{B}{\\|B\\|}\n",
"$$"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "b8be02be-3ac3-4e5f-a450-c53f05781ab4",
"metadata": {
"ExecutionIndicator": {
"show": true
},
"execution": {
"iopub.execute_input": "2024-07-17T02:24:38.945105Z",
"iopub.status.busy": "2024-07-17T02:24:38.944403Z",
"iopub.status.idle": "2024-07-17T02:24:38.948117Z",
"shell.execute_reply": "2024-07-17T02:24:38.947698Z",
"shell.execute_reply.started": "2024-07-17T02:24:38.945085Z"
},
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0.802726686000824\n"
]
}
],
"source": [
"res = torch.mm(A / A.norm(dim=1), B.T / B.norm(dim=1))\n",
"print(res.item())"
]
},
{
"cell_type": "markdown",
"id": "988acff0-e6b5-41db-92d6-8f175dd3e272",
"metadata": {
"tags": []
},
"source": [
"### 3.3 PyTorch Function Call"
]
},
{
"cell_type": "markdown",
"id": "a61b4871-4039-4c6e-b5ee-f66a12156be9",
"metadata": {},
"source": [
"In practice, the most convinient way is directly use *cosine_similarity()* in torch.nn.functional:"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "1ac4012e-b90a-4e60-97b8-e42636fde1c9",
"metadata": {
"ExecutionIndicator": {
"show": true
},
"execution": {
"iopub.execute_input": "2024-07-17T02:24:55.804298Z",
"iopub.status.busy": "2024-07-17T02:24:55.803810Z",
"iopub.status.idle": "2024-07-17T02:24:55.807551Z",
"shell.execute_reply": "2024-07-17T02:24:55.807146Z",
"shell.execute_reply.started": "2024-07-17T02:24:55.804278Z"
},
"tags": []
},
"outputs": [
{
"data": {
"text/plain": [
"0.802726686000824"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import torch.nn.functional as F\n",
"\n",
"F.cosine_similarity(A, B).item()"
]
},
{
"cell_type": "markdown",
"id": "f4ab87cc",
"metadata": {},
"source": [
"## 4. Inner Product/Dot Product"
]
},
{
"cell_type": "markdown",
"id": "e3c025ab",
"metadata": {},
"source": [
"Coordinate definition:\n",
"$$A\\cdot B = \\sum_{i=1}^{i=n}A_i B_i$$\n",
"\n",
"Geometric definition:\n",
"$$A\\cdot B = \\|A\\|\\|B\\|\\cos(\\theta)$$"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "f0291d42",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"68.0"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"dot_prod = A @ B.T\n",
"dot_prod.item()"
]
},
{
"cell_type": "markdown",
"id": "33099a2e",
"metadata": {},
"source": [
"### Relationship with Cosine similarity"
]
},
{
"cell_type": "markdown",
"id": "2790e183",
"metadata": {},
"source": [
"For computing the distance/similarity between two vectors, dot product and Cos similarity are closely related. Cos similarity only cares about the angle difference (because it is normalized by the product of two vectors' magnitude), while dot product takes both magnitude and angle into consideration. So the two metrics are preferred in different use cases.\n",
"\n",
"The BGE series models already normalized the output embedding vector to have the magnitude of 1. Thus using dot product and cos similarity will have the same result."
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "e0f40534",
"metadata": {},
"outputs": [],
"source": [
"from FlagEmbedding import FlagModel\n",
"\n",
"model = FlagModel('BAAI/bge-large-en-v1.5',\n",
" query_instruction_for_retrieval=\"Represent this sentence for searching relevant passages:\",\n",
" use_fp16=True)"
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "78445a86",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"1.0"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sentence = \"I am very interested in natural language processing\"\n",
"embedding = torch.tensor(model.encode(sentence))\n",
"torch.norm(embedding).item()"
]
},
{
"cell_type": "markdown",
"id": "9e1822ee",
"metadata": {},
"source": [
"## 5. Examples"
]
},
{
"cell_type": "markdown",
"id": "6c665e3a",
"metadata": {},
"source": [
"Now we've learned the mechanism of different types of similarity. Let's look at a real example."
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "73012cbb",
"metadata": {},
"outputs": [],
"source": [
"sentence_1 = \"I will watch a show tonight\"\n",
"sentence_2 = \"I will show you my watch tonight\"\n",
"sentence_3 = \"I'm going to enjoy a performance this evening\""
]
},
{
"cell_type": "markdown",
"id": "3cb79a47",
"metadata": {},
"source": [
"It's clear to us that in sentence 1, 'watch' is a verb and 'show' is a noun. \n",
"\n",
"But in sentence 2, 'show' is a verb and 'watch' is a noun, which leads to different meaning of the two sentences.\n",
"\n",
"While sentence 3 has very similar meaning to sentence 1."
]
},
{
"cell_type": "markdown",
"id": "dc44dee9",
"metadata": {},
"source": [
"Now let's see how does different similarity metrics tell us the relationship of the sentences."
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "98bfcc6d",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0.625\n",
"0.07692307692307693\n"
]
}
],
"source": [
"print(jaccard_similarity(sentence_1, sentence_2))\n",
"print(jaccard_similarity(sentence_1, sentence_3))"
]
},
{
"cell_type": "markdown",
"id": "b7e4cd15",
"metadata": {},
"source": [
"The results show that sentence 1 and 2 (0.625) are way more similar than sentence 1 and 3 (0.077), which indicate the opposite conclusion compare to what we have made."
]
},
{
"cell_type": "markdown",
"id": "cff73692",
"metadata": {},
"source": [
"Now let's first get the embeddings of these sentences."
]
},
{
"cell_type": "code",
"execution_count": 18,
"id": "426c0b42",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"torch.Size([1, 1024])\n"
]
}
],
"source": [
"embeddings = torch.from_numpy(model.encode([sentence_1, sentence_2, sentence_3]))\n",
"embedding_1 = embeddings[0].view(1, -1)\n",
"embedding_2 = embeddings[1].view(1, -1)\n",
"embedding_3 = embeddings[2].view(1, -1)\n",
"\n",
"print(embedding_1.shape)"
]
},
{
"cell_type": "markdown",
"id": "63fe1b31",
"metadata": {},
"source": [
"Then let's compute the Euclidean distance:"
]
},
{
"cell_type": "code",
"execution_count": 19,
"id": "d9bb35cf",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0.714613139629364\n",
"0.5931472182273865\n"
]
}
],
"source": [
"euc_dist1_2 = torch.cdist(embedding_1, embedding_2, p=2).item()\n",
"euc_dist1_3 = torch.cdist(embedding_1, embedding_3, p=2).item()\n",
"print(euc_dist1_2)\n",
"print(euc_dist1_3)"
]
},
{
"cell_type": "markdown",
"id": "402e6ea8",
"metadata": {},
"source": [
"Then, let's see the cosine similarity:"
]
},
{
"cell_type": "code",
"execution_count": 20,
"id": "29e70bbc",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0.7446640729904175\n",
"0.8240882158279419\n"
]
}
],
"source": [
"cos_dist1_2 = F.cosine_similarity(embedding_1, embedding_2).item()\n",
"cos_dist1_3 = F.cosine_similarity(embedding_1, embedding_3).item()\n",
"print(cos_dist1_2)\n",
"print(cos_dist1_3)"
]
},
{
"cell_type": "markdown",
"id": "c353d8cc",
"metadata": {},
"source": [
"Using embedding, we can get the correct result different from Jaccard similarity that sentence 1 and 2 should be more similar than sentence 1 and 3 using either Euclidean distance or cos similarity as the metric."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.13"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
docs/source/tutorial/2_Metrics/2.2.ipynb
+472
View File
@@ -0,0 +1,472 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Evaluation Metrics"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In this tutorial, we'll cover a list of metrics that are widely used for evaluating embedding model's performance."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 0. Preparation"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%pip install numpy scikit-learn"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Suppose we have a corpus with document ids from 0 - 30. \n",
"- `ground_truth` contains the actual relevant document ids to each query.\n",
"- `results` contains the search results of each query by some retrieval system."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"\n",
"ground_truth = [\n",
" [11, 1, 7, 17, 21],\n",
" [ 4, 16, 1],\n",
" [26, 10, 22, 8],\n",
"]\n",
"\n",
"results = [\n",
" [11, 1, 17, 7, 21, 8, 0, 28, 9, 20],\n",
" [16, 1, 6, 18, 3, 4, 25, 19, 8, 14],\n",
" [24, 10, 26, 2, 8, 28, 4, 23, 13, 21],\n",
"]"
]
},
{
"cell_type": "code",
"execution_count": 63,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([ 0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 13, 14, 16, 17, 18, 19,\n",
" 21, 22, 24, 25, 26, 28])"
]
},
"execution_count": 63,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.intersect1d(ground_truth, results)"
]
},
{
"cell_type": "code",
"execution_count": 65,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[1, 1, 1, 1, 1, 1, 1, 1, 1, 0],\n",
" [1, 1, 1, 1, 1, 1, 1, 1, 1, 0],\n",
" [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]])"
]
},
"execution_count": 65,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.isin(ground_truth, results).astype(int)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"And we are interested in the following cutoffs:"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"cutoffs = [1, 5, 10]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In this tutorial, we will use the above small example to show how different metrics evaluate the retrieval system's quality."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. Recall"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Recall represents the model's capability of correctly predicting positive instances from all the actual positive samples in the dataset.\n",
"\n",
"$$\\textbf{Recall}=\\frac{\\text{True Positives}}{\\text{True Positives}+\\text{False Negatives}}$$\n",
"\n",
"to write it in the form of information retrieval, which is the ratio of relevant documents retrieved to the total number of relevant documents in the corpus. In practice, we usually make the denominator to be the minimum between the current cutoff (usually 1, 5, 10, 100, etc) and the total number of relevant documents in the corpus:\n",
"\n",
"$$\\textbf{Recall}=\\frac{|\\text{\\{Relevant docs\\}}\\cap\\text{\\{Retrieved docs\\}}|}{\\text{min}(|\\text{\\{Retrieved docs\\}}|, |\\text{\\{Relevant docs\\}}|)}$$"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"def calc_recall(preds, truths, cutoffs):\n",
" recalls = np.zeros(len(cutoffs))\n",
" for text, truth in zip(preds, truths):\n",
" for i, c in enumerate(cutoffs):\n",
" hits = np.intersect1d(truth, text[:c])\n",
" recalls[i] += len(hits) / max(min(c, len(truth)), 1)\n",
" recalls /= len(preds)\n",
" return recalls"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"recall@1: 0.6666666666666666\n",
"recall@5: 0.8055555555555555\n",
"recall@10: 0.9166666666666666\n"
]
}
],
"source": [
"recalls = calc_recall(results, ground_truth, cutoffs)\n",
"for i, c in enumerate(cutoffs):\n",
" print(f\"recall@{c}: {recalls[i]}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2. MRR"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Mean Reciprocal Rank ([MRR](https://en.wikipedia.org/wiki/Mean_reciprocal_rank)) is a widely used metric in information retrieval to evaluate the effectiveness of a system. It measures the rank position of the first relevant result in a list of search results.\n",
"\n",
"$$MRR=\\frac{1}{|Q|}\\sum_{i=1}^{|Q|}\\frac{1}{rank_i}$$\n",
"\n",
"where \n",
"- $|Q|$ is the total number of queries.\n",
"- $rank_i$ is the rank position of the first relevant document of the i-th query."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"def calc_MRR(preds, truth, cutoffs):\n",
" mrr = [0 for _ in range(len(cutoffs))]\n",
" for pred, t in zip(preds, truth):\n",
" for i, c in enumerate(cutoffs):\n",
" for j, p in enumerate(pred):\n",
" if j < c and p in t:\n",
" mrr[i] += 1/(j+1)\n",
" break\n",
" mrr = [k/len(preds) for k in mrr]\n",
" return mrr"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"MRR@1: 0.6666666666666666\n",
"MRR@5: 0.8333333333333334\n",
"MRR@10: 0.8333333333333334\n"
]
}
],
"source": [
"mrr = calc_MRR(results, ground_truth, cutoffs)\n",
"for i, c in enumerate(cutoffs):\n",
" print(f\"MRR@{c}: {mrr[i]}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3. nDCG"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Normalized Discounted Cumulative Gain (nDCG) measures the quality of a ranked list of search results by considering both the position of the relevant documents and their graded relevance scores. The calculation of nDCG involves two main steps:\n",
"\n",
"1. Discounted cumulative gain (DCG) measures the ranking quality in retrieval tasks.\n",
"\n",
"$$DCG_p=\\sum_{i=1}^p\\frac{2^{rel_i}-1}{\\log_2(i+1)}$$\n",
"\n",
"2. Normalized by ideal DCG to make it comparable across queries.\n",
"$$nDCG_p=\\frac{DCG_p}{IDCG_p}$$\n",
"where $IDCG$ is the maximum possible DCG for a given set of documents, assuming they are perfectly ranked in order of relevance."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [],
"source": [
"pred_hard_encodings = []\n",
"for pred, label in zip(results, ground_truth):\n",
" pred_hard_encoding = list(np.isin(pred, label).astype(int))\n",
" pred_hard_encodings.append(pred_hard_encoding)"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"nDCG@1: 0.0\n",
"nDCG@5: 0.3298163165186628\n",
"nDCG@10: 0.5955665344840209\n"
]
}
],
"source": [
"from sklearn.metrics import ndcg_score\n",
"\n",
"for i, c in enumerate(cutoffs):\n",
" nDCG = ndcg_score(pred_hard_encodings, results, k=c)\n",
" print(f\"nDCG@{c}: {nDCG}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 4. Precision"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Precision \n",
"\n",
"$$\\textbf{Recall}=\\frac{\\text{True Positives}}{\\text{True Positives}+\\text{False Positive}}$$\n",
"\n",
"in information retrieval, it's the ratio of relevant documents retrieved to the totoal number of documents retrieved:\n",
"\n",
"$$\\textbf{Recall}=\\frac{|\\text{\\{Relevant docs\\}}\\cap\\text{\\{Retrieved docs\\}}|}{|\\text{\\{Retrieved docs\\}}|}$$"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
"def calc_precision(preds, truths, cutoffs):\n",
" prec = np.zeros(len(cutoffs))\n",
" for text, truth in zip(preds, truths):\n",
" for i, c in enumerate(cutoffs):\n",
" hits = np.intersect1d(truth, text[:c])\n",
" prec[i] += len(hits) / c\n",
" prec /= len(preds)\n",
" return prec"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"precision@1: 0.6666666666666666\n",
"precision@5: 0.6666666666666666\n",
"precision@10: 0.3666666666666667\n"
]
}
],
"source": [
"precisions = calc_precision(results, ground_truth, cutoffs)\n",
"for i, c in enumerate(cutoffs):\n",
" print(f\"precision@{c}: {precisions[i]}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 5. MAP"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Mean Average Precision (MAP) measures the effectiveness of a system at returning relevant documents across multiple queries. \n",
"\n",
"First, Average Precision (AP) evals how well relevant documents are ranked within the retrieved documents. It's computed by averaging the precision values for each position of relevant document in the ranking of all the retrieved documents:\n",
"\n",
"$$\\textbf{AP}=\\frac{\\sum_{k=1}^{M}\\text{Relevance}(k) \\times \\text{Precision}(k)}{|\\{\\text{Relevant Docs}\\}|}$$\n",
"\n",
"where \n",
"- $M$ is the total number of documents retrieved.\n",
"- $\\text{Relevance}(k)$ is a binary value, indicating whether document at position $k$ is relevant (=1) or not (=0).\n",
"- $\\text{Precision}(k)$ is the precision when considering only top $k$ retrieved items."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Then calculate the average AP across multiple queries to get the MAP:\n",
"\n",
"$$\\textbf{MAP}=\\frac{1}{N}\\sum_{i=1}^{N}\\text{AP}_i$$\n",
"\n",
"where\n",
"- $N$ is the total number of queries.\n",
"- $\\text{AP}_i$ is the average precision of the $i^{th}$ query."
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [],
"source": [
"def calc_AP(encoding):\n",
" rel = 0\n",
" precs = 0.0\n",
" for k, hit in enumerate(encoding, start=1):\n",
" if hit == 1:\n",
" rel += 1\n",
" precs += rel/k\n",
"\n",
" return 0 if rel == 0 else precs/rel"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [],
"source": [
"def calc_MAP(encodings, cutoffs):\n",
" res = []\n",
" for c in cutoffs:\n",
" ap_sum = 0.0\n",
" for encoding in encodings:\n",
" ap_sum += calc_AP(encoding[:c])\n",
" res.append(ap_sum/len(encodings))\n",
" \n",
" return res"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"MAP@1: 0.6666666666666666\n",
"MAP@5: 0.862962962962963\n",
"MAP@10: 0.8074074074074075\n"
]
}
],
"source": [
"maps = calc_MAP(pred_hard_encodings, cutoffs)\n",
"for i, c in enumerate(cutoffs):\n",
" print(f\"MAP@{c}: {maps[i]}\")"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "test",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.4"
}
},
"nbformat": 4,
"nbformat_minor": 2
}