chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,38 @@
|
||||
Information Retrieval
|
||||
=====================
|
||||
|
||||
What is Information Retrieval?
|
||||
------------------------------
|
||||
|
||||
Simply put, Information Retrieval (IR) is the science of searching and retrieving information from a large collection of data based on a user's query.
|
||||
The goal of an IR system is not just to return a list of documents but to ensure that the most relevant ones appear at the top of the results.
|
||||
|
||||
A very straightforward example of IR is library catalog. One wants to find the book that best matches the query, but there are thousands or millions of books on the shelf.
|
||||
The library's catalog system helps you find the best matches based on your search terms.
|
||||
In modern digital world, search engines and databases work in a similar way, using sophisticated algorithms and models to retrieve, rank and return the most relevant results.
|
||||
And the resource categories are expanding from text to more modalities such as images, videos, 3D objects, music, etc.
|
||||
|
||||
IR and Embedding Model
|
||||
----------------------
|
||||
|
||||
Traditional IR methods, like TF-IDF and BM25, rely on statistical and heuristic techniques to rank documents based on term frequency and document relevance.
|
||||
These methods are efficient and effective for keyword-based search but often struggle with understanding the deeper context or semantics of the text.
|
||||
|
||||
.. seealso::
|
||||
|
||||
Take a very simple example with two sentences:
|
||||
|
||||
.. code:: python
|
||||
|
||||
sentence_1 = "watch a play"
|
||||
sentence_2 = "play with a watch"
|
||||
|
||||
Sentence 1 means going for a show/performance, which has watch as a verb and play as a noun.
|
||||
|
||||
However sentence 2 means someone is interacting with a timepiece on wrist, which has play as a verb and watch as a noun.
|
||||
|
||||
These two sentences could be regard as very similar to each other when using the traditional IR methods though they actually have totally different semantic meaning.
|
||||
Then how could we solve this? The best answer up until now is embedding models.
|
||||
|
||||
Embedding models have revolutionized IR by representing text as dense vectors in a high-dimensional space, capturing the semantic meaning of words, sentences, or even entire documents.
|
||||
This allows for more sophisticated search capabilities, such as semantic search, where results are ranked based on meaning rather than simple keyword matching.
|
||||
@@ -0,0 +1,39 @@
|
||||
Embedder
|
||||
========
|
||||
|
||||
.. tip::
|
||||
|
||||
If you are already familiar with the concepts, take a look at the :doc:`BGE models <../bge/index>`!
|
||||
|
||||
Embedder, or embedding model, bi-encoder, is a model designed to convert data, usually text, codes, or images, into sparse or dense numerical vectors (embeddings) in a high dimensional vector space.
|
||||
These embeddings capture the semantic meaning or key features of the input, which enable efficient comparison and analysis.
|
||||
|
||||
A very famous demonstration is the example from `word2vec <https://arxiv.org/abs/1301.3781>`_. It shows how word embeddings capture semantic relationships through vector arithmetic:
|
||||
|
||||
.. image:: ../_static/img/word2vec.png
|
||||
:width: 500
|
||||
:align: center
|
||||
|
||||
Nowadays, embedders are capable of mapping sentences and even passages into vector space.
|
||||
They are widely used in real world tasks such as retrieval, clustering, etc.
|
||||
In the era of LLMs, embedding models play a pivot role in RAG, enables LLMs to access and integrate relevant context from vast external datasets.
|
||||
|
||||
|
||||
Sparse Vector
|
||||
-------------
|
||||
|
||||
Sparse vectors usually have structure of high dimensionality with only a few non-zero values, which usually effective for tasks like keyword matching.
|
||||
Typically, though not always, the number of dimensions in sparse vectors corresponds to the different tokens present in the language.
|
||||
Each dimension is assigned a value representing the token's relative importance within the document.
|
||||
Some well known algorithms for sparse vector embedding includes `bag-of-words <https://en.wikipedia.org/wiki/Bag-of-words_model>`_, `TF-IDF <https://en.wikipedia.org/wiki/Tf%E2%80%93idf>`_, `BM25 <https://en.wikipedia.org/wiki/Okapi_BM25>`_, etc.
|
||||
Sparse vector embeddings have great ability to extract the information of key terms and their corresponding importance within documents.
|
||||
|
||||
Dense Vector
|
||||
------------
|
||||
|
||||
Dense vectors typically use neural networks to map words, sentences, and passages into a fixed dimension latent vector space.
|
||||
Then we can compare the similarity between two objects using certain metrics like Euclidean distance or Cos similarity.
|
||||
Those vectors can represent deeper meaning of the sentences.
|
||||
Thus we can distinguish sentences using similar words but actually have different meaning.
|
||||
And also understand different ways in speaking and writing that express the same thing.
|
||||
Dense vector embeddings, instead of keywords counting and matching, directly capture the semantics.
|
||||
@@ -0,0 +1,31 @@
|
||||
Introduction
|
||||
============
|
||||
|
||||
BGE builds one-stop retrieval toolkit for search and RAG. We provide inference, evaluation, and fine-tuning for embedding models and reranker.
|
||||
|
||||
.. figure:: ../_static/img/RAG_pipeline.png
|
||||
:width: 700
|
||||
:align: center
|
||||
|
||||
BGE embedder and reranker in an RAG pipeline. `Source <https://safjan.com/images/retrieval_augmented_generation/RAG.png>`_
|
||||
|
||||
Quickly get started with:
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
:caption: Start
|
||||
|
||||
overview
|
||||
installation
|
||||
quick_start
|
||||
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
:caption: Concept
|
||||
|
||||
IR
|
||||
embedder
|
||||
reranker
|
||||
similarity
|
||||
retrieval_demo
|
||||
@@ -0,0 +1,48 @@
|
||||
:github_url: https://github.com/AI4Finance-Foundation/FinRL
|
||||
|
||||
Installation
|
||||
============
|
||||
|
||||
Using pip:
|
||||
----------
|
||||
|
||||
If you do not need to finetune the models, you can install the package without the finetune dependency:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
pip install -U FlagEmbedding
|
||||
|
||||
If you want to finetune the models, you can install the package with the finetune dependency:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
pip install -U FlagEmbedding[finetune]
|
||||
|
||||
|
||||
Install from sources:
|
||||
---------------------
|
||||
|
||||
Clone the repository and install
|
||||
|
||||
.. code:: bash
|
||||
|
||||
git clone https://github.com/FlagOpen/FlagEmbedding.git
|
||||
cd FlagEmbedding
|
||||
# If you do not need to finetune the models, you can install the package without the finetune dependency:
|
||||
pip install .
|
||||
# If you want to finetune the models, install the package with the finetune dependency:
|
||||
pip install .[finetune]
|
||||
|
||||
For development in editable mode:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
# If you do not need to finetune the models, you can install the package without the finetune dependency:
|
||||
pip install -e .
|
||||
# If you want to finetune the models, install the package with the finetune dependency:
|
||||
pip install -e .[finetune]
|
||||
|
||||
PyTorch-CUDA
|
||||
------------
|
||||
|
||||
If you want to use CUDA GPUs during inference and finetuning, please install appropriate version of `PyTorch <https://pytorch.org/get-started/locally/>`_ with CUDA support.
|
||||
@@ -0,0 +1,17 @@
|
||||
Overview
|
||||
========
|
||||
|
||||
Our repository provides well-structured `APIs <https://github.com/FlagOpen/FlagEmbedding/tree/master/FlagEmbedding>`_ for the inference, evaluation, and fine-tuning of BGE series models.
|
||||
Besides that, there are abundant resources of and for users to quickly get a hands-on experience.
|
||||
|
||||
.. figure:: https://raw.githubusercontent.com/FlagOpen/FlagEmbedding/refs/heads/master/imgs/projects.png
|
||||
:width: 700
|
||||
:align: center
|
||||
|
||||
Structure of contents in our `repo <https://github.com/FlagOpen/FlagEmbedding>`_
|
||||
|
||||
Our repository provides well-structured contents for information retrieval and RAG:
|
||||
|
||||
- The core `APIs <../API>`_ for embedding models' inference, evaluation, and fine-tuning.
|
||||
- Hands-on `examples <https://github.com/FlagOpen/FlagEmbedding/tree/master/examples>`_ for the three mentioned use cases.
|
||||
- Detailed `tutorials <https://github.com/FlagOpen/FlagEmbedding/tree/master/Tutorials>`_ covering topics in retrieval to help you learn from scratch.
|
||||
@@ -0,0 +1,34 @@
|
||||
Quick Start
|
||||
===========
|
||||
|
||||
First, load one of the BGE embedding model:
|
||||
|
||||
.. code:: python
|
||||
|
||||
from FlagEmbedding import FlagAutoModel
|
||||
|
||||
model = FlagAutoModel.from_finetuned('BAAI/bge-base-en-v1.5')
|
||||
|
||||
.. tip::
|
||||
|
||||
If there's difficulty connecting to Hugging Face, you can use the `HF mirror <https://hf-mirror.com/>`_ instead.
|
||||
|
||||
.. code:: bash
|
||||
|
||||
export HF_ENDPOINT=https://hf-mirror.com
|
||||
|
||||
Then, feed some sentences to the model and get their embeddings:
|
||||
|
||||
.. code:: python
|
||||
|
||||
sentences_1 = ["I love NLP", "I love machine learning"]
|
||||
sentences_2 = ["I love BGE", "I love text retrieval"]
|
||||
embeddings_1 = model.encode(sentences_1)
|
||||
embeddings_2 = model.encode(sentences_2)
|
||||
|
||||
Once we get the embeddings, we can compute similarity by inner product:
|
||||
|
||||
.. code:: python
|
||||
|
||||
similarity = embeddings_1 @ embeddings_2.T
|
||||
print(similarity)
|
||||
@@ -0,0 +1,22 @@
|
||||
Reranker
|
||||
========
|
||||
|
||||
.. tip::
|
||||
|
||||
If you are already familiar with the concepts, take a look at the :doc:`BGE rerankers <../bge/index>`!
|
||||
|
||||
Reranker, or Cross-Encoder, is a model that refines the ranking of candidate pairs (e.g., query-document pairs) by jointly encoding and scoring them.
|
||||
|
||||
Typically, we use embedder as a Bi-Encoder. It first computes the embeddings of two input sentences, then compute their similarity using metrics such as cosine similarity or Euclidean distance.
|
||||
Whereas a reranker takes two sentences at the same time and directly computer a score representing their similarity.
|
||||
|
||||
The following figure shows their difference:
|
||||
|
||||
.. figure:: https://raw.githubusercontent.com/UKPLab/sentence-transformers/master/docs/img/Bi_vs_Cross-Encoder.png
|
||||
:width: 500
|
||||
:align: center
|
||||
|
||||
Bi-Encoder & Cross-Encoder (from Sentence Transformers)
|
||||
|
||||
Although Cross-Encoder usually has better performances than Bi-Encoder, it is extremly time consuming to use Cross-Encoder if we have a great amount of data.
|
||||
Thus a widely accepted approach is to use a Bi-Encoder for initial retrieval (e.g., selecting the top 100 candidates from 100,000 sentences) and then refine the ranking of the selected candidates using a Cross-Encoder for more accurate results.
|
||||
@@ -0,0 +1,472 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Retrieval Demo"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"In this tutorial, we will show how to use BGE models on a text retrieval task in 5 minutes."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Step 0: Preparation"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"First, install FlagEmbedding in the environment."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%pip install -U FlagEmbedding"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Below is a super tiny courpus with only 10 sentences, which will be the dataset we use.\n",
|
||||
"\n",
|
||||
"Each sentence is a concise discription of a famous people in specific domain."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"corpus = [\n",
|
||||
" \"Michael Jackson was a legendary pop icon known for his record-breaking music and dance innovations.\",\n",
|
||||
" \"Fei-Fei Li is a professor in Stanford University, revolutionized computer vision with the ImageNet project.\",\n",
|
||||
" \"Brad Pitt is a versatile actor and producer known for his roles in films like 'Fight Club' and 'Once Upon a Time in Hollywood.'\",\n",
|
||||
" \"Geoffrey Hinton, as a foundational figure in AI, received Turing Award for his contribution in deep learning.\",\n",
|
||||
" \"Eminem is a renowned rapper and one of the best-selling music artists of all time.\",\n",
|
||||
" \"Taylor Swift is a Grammy-winning singer-songwriter known for her narrative-driven music.\",\n",
|
||||
" \"Sam Altman leads OpenAI as its CEO, with astonishing works of GPT series and pursuing safe and beneficial AI.\",\n",
|
||||
" \"Morgan Freeman is an acclaimed actor famous for his distinctive voice and diverse roles.\",\n",
|
||||
" \"Andrew Ng spread AI knowledge globally via public courses on Coursera and Stanford University.\",\n",
|
||||
" \"Robert Downey Jr. is an iconic actor best known for playing Iron Man in the Marvel Cinematic Universe.\",\n",
|
||||
"]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We want to know which one of these people could be an expert of neural network and who he/she is. \n",
|
||||
"\n",
|
||||
"Thus we generate the following query:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"query = \"Who could be an expert of neural network?\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Step 1: Text -> Embedding"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"First, let's use a [BGE embedding model](https://huggingface.co/BAAI/bge-base-en-v1.5) to create sentence embedding for the corpus."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 15,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from FlagEmbedding import FlagModel\n",
|
||||
"\n",
|
||||
"# get the BGE embedding model\n",
|
||||
"model = FlagModel('BAAI/bge-base-en-v1.5',\n",
|
||||
" query_instruction_for_retrieval=\"Represent this sentence for searching relevant passages:\",\n",
|
||||
" use_fp16=True)\n",
|
||||
"\n",
|
||||
"# get the embedding of the query and corpus\n",
|
||||
"corpus_embeddings = model.encode(corpus)\n",
|
||||
"query_embedding = model.encode(query)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"The embedding of each sentence is a vector with length 768. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 16,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"shape of the query embedding: (768,)\n",
|
||||
"shape of the corpus embeddings: (10, 768)\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"print(\"shape of the query embedding: \", query_embedding.shape)\n",
|
||||
"print(\"shape of the corpus embeddings:\", corpus_embeddings.shape)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Run the following print line to take a look at the first 10 elements of the query embedding vector."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 17,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"[-0.00790005 -0.00683443 -0.00806659 0.00756918 0.04374858 0.02838556\n",
|
||||
" 0.02357143 -0.02270943 -0.03611493 -0.03038301]\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"print(query_embedding[:10])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Step 2: Calculate Similarity"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Now, we have the embeddings of the query and the corpus. The next step is to calculate the similarity between the query and each sentence in the corpus. Here we use the dot product/inner product as our similarity metric."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 18,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"[0.39290053 0.6031525 0.32672375 0.6082418 0.39446455 0.35350388\n",
|
||||
" 0.4626108 0.40196604 0.5284606 0.36792332]\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"sim_scores = query_embedding @ corpus_embeddings.T\n",
|
||||
"print(sim_scores)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"The result is a list of score representing the query's similarity to: [sentence 0, sentence 1, sentence 2, ...]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Step 3: Ranking"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"After we have the similarity score of the query to each sentence in the corpus, we can rank them from large to small."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 19,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"[3, 1, 8, 6, 7, 4, 0, 9, 5, 2]\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# get the indices in sorted order\n",
|
||||
"sorted_indices = sorted(range(len(sim_scores)), key=lambda k: sim_scores[k], reverse=True)\n",
|
||||
"print(sorted_indices)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Now from the ranking, the sentence with index 3 is the best answer to our query \"Who could be an expert of neural network?\"\n",
|
||||
"\n",
|
||||
"And that person is Geoffrey Hinton!"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 20,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Geoffrey Hinton, as a foundational figure in AI, received Turing Award for his contribution in deep learning.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"print(corpus[3])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"According to the order of indecies, we can print out the ranking of people that our little retriever got."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 21,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Score of 0.608: \"Geoffrey Hinton, as a foundational figure in AI, received Turing Award for his contribution in deep learning.\"\n",
|
||||
"Score of 0.603: \"Fei-Fei Li is a professor in Stanford University, revolutionized computer vision with the ImageNet project.\"\n",
|
||||
"Score of 0.528: \"Andrew Ng spread AI knowledge globally via public courses on Coursera and Stanford University.\"\n",
|
||||
"Score of 0.463: \"Sam Altman leads OpenAI as its CEO, with astonishing works of GPT series and pursuing safe and beneficial AI.\"\n",
|
||||
"Score of 0.402: \"Morgan Freeman is an acclaimed actor famous for his distinctive voice and diverse roles.\"\n",
|
||||
"Score of 0.394: \"Eminem is a renowned rapper and one of the best-selling music artists of all time.\"\n",
|
||||
"Score of 0.393: \"Michael Jackson was a legendary pop icon known for his record-breaking music and dance innovations.\"\n",
|
||||
"Score of 0.368: \"Robert Downey Jr. is an iconic actor best known for playing Iron Man in the Marvel Cinematic Universe.\"\n",
|
||||
"Score of 0.354: \"Taylor Swift is a Grammy-winning singer-songwriter known for her narrative-driven music.\"\n",
|
||||
"Score of 0.327: \"Brad Pitt is a versatile actor and producer known for his roles in films like 'Fight Club' and 'Once Upon a Time in Hollywood.'\"\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# iteratively print the score and corresponding sentences in descending order\n",
|
||||
"\n",
|
||||
"for i in sorted_indices:\n",
|
||||
" print(f\"Score of {sim_scores[i]:.3f}: \\\"{corpus[i]}\\\"\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"From the ranking, not surprisingly, the similarity scores of the query and the discriptions of Geoffrey Hinton and Fei-Fei Li is way higher than others, following by those of Andrew Ng and Sam Altman. \n",
|
||||
"\n",
|
||||
"While the key phrase \"neural network\" in the query does not appear in any of those discriptions, the BGE embedding model is still powerful enough to get the semantic meaning of query and corpus well."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Step 4: Evaluate"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We've seen the embedding model performed pretty well on the \"neural network\" query. What about the more general quality?\n",
|
||||
"\n",
|
||||
"Let's generate a very small dataset of queries and corresponding ground truth answers. Note that the ground truth answers are the indices of sentences in the corpus."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 22,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"queries = [\n",
|
||||
" \"Who could be an expert of neural network?\",\n",
|
||||
" \"Who might had won Grammy?\",\n",
|
||||
" \"Won Academy Awards\",\n",
|
||||
" \"One of the most famous female singers.\",\n",
|
||||
" \"Inventor of AlexNet\",\n",
|
||||
"]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 23,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"ground_truth = [\n",
|
||||
" [1, 3],\n",
|
||||
" [0, 4, 5],\n",
|
||||
" [2, 7, 9],\n",
|
||||
" [5],\n",
|
||||
" [3],\n",
|
||||
"]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Here we repeat the steps we covered above to get the predicted ranking of each query."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 24,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[[3, 1, 8, 6, 7, 4, 0, 9, 5, 2],\n",
|
||||
" [5, 0, 3, 4, 1, 9, 7, 2, 6, 8],\n",
|
||||
" [3, 2, 7, 5, 9, 0, 1, 4, 6, 8],\n",
|
||||
" [5, 0, 4, 7, 1, 9, 2, 3, 6, 8],\n",
|
||||
" [3, 1, 8, 6, 0, 7, 5, 9, 4, 2]]"
|
||||
]
|
||||
},
|
||||
"execution_count": 24,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# use bge model to generate embeddings for all the queries\n",
|
||||
"queries_embedding = model.encode(queries)\n",
|
||||
"# compute similarity scores\n",
|
||||
"scores = queries_embedding @ corpus_embeddings.T\n",
|
||||
"# get he final rankings\n",
|
||||
"rankings = [sorted(range(len(sim_scores)), key=lambda k: sim_scores[k], reverse=True) for sim_scores in scores]\n",
|
||||
"rankings"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Mean Reciprocal Rank ([MRR](https://en.wikipedia.org/wiki/Mean_reciprocal_rank)) is a widely used metric in information retrieval to evaluate the effectiveness of a system. Here we use that to have a very rough idea how our system performs."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 25,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def MRR(preds, labels, cutoffs):\n",
|
||||
" mrr = [0 for _ in range(len(cutoffs))]\n",
|
||||
" for pred, label in zip(preds, labels):\n",
|
||||
" for i, c in enumerate(cutoffs):\n",
|
||||
" for j, index in enumerate(pred):\n",
|
||||
" if j < c and index in label:\n",
|
||||
" mrr[i] += 1/(j+1)\n",
|
||||
" break\n",
|
||||
" mrr = [k/len(preds) for k in mrr]\n",
|
||||
" return mrr"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We choose to use 1 and 5 as our cutoffs, with the result of 0.8 and 0.9 respectively."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 26,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"MRR@1: 0.8\n",
|
||||
"MRR@5: 0.9\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"cutoffs = [1, 5]\n",
|
||||
"mrrs = MRR(rankings, ground_truth, cutoffs)\n",
|
||||
"for i, c in enumerate(cutoffs):\n",
|
||||
" print(f\"MRR@{c}: {mrrs[i]}\")"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "base",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.13"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
@@ -0,0 +1,60 @@
|
||||
Similarity
|
||||
==========
|
||||
|
||||
A primary goal of retrieval is to find the most relevant documents in response to a user's query.
|
||||
One of the core components of this process is measuring similarity between the query and candidates.
|
||||
Similarity metrics quantify how closely related two pieces of data are, and guide the retrieval system in ranking results.
|
||||
|
||||
Jaccard Similarity
|
||||
------------------
|
||||
|
||||
.. math::
|
||||
|
||||
J(A,B)=\frac{|A\cap B|}{|A\cup B|}
|
||||
|
||||
The Jaccard similarity or Jaccard index is commonly used for set-based similarity, particularly in binary data (e.g., whether a term appears in a document or not).
|
||||
It is calculated as the size of the intersection of two sets divided by the size of their union.
|
||||
In information retrieval, it's often used to compare sets of keywords or phrases, with higher values indicating more similarity.
|
||||
|
||||
Euclidean Distance
|
||||
------------------
|
||||
|
||||
.. math::
|
||||
|
||||
d(A, B) = \|A-B\|_2 = \sqrt{\sum_{i=1}^n (A_i-B_i)^2}
|
||||
|
||||
Euclidean distance measures the straight-line distance between two points in a vector space.
|
||||
In IR, this can be used to assess the difference between document or query vectors.
|
||||
A smaller distance indicates greater similarity.
|
||||
This metric is intuitive but can sometimes be sensitive to the scale of the data, especially in high-dimensional spaces like text embeddings.
|
||||
|
||||
Cosine Similarity
|
||||
-----------------
|
||||
|
||||
.. math::
|
||||
|
||||
\cos(\theta)=\frac{A\cdot B}{\|A\|\|B\|}
|
||||
|
||||
Cosine similarity is one of the most widely used metrics in information retrieval, especially for text.
|
||||
It measures the cosine of the angle between two vectors in a multi-dimensional space (typically representing term frequency vectors of documents and queries).
|
||||
If the cosine similarity is closer to 1, the vectors are more similar.
|
||||
A value of 0 indicates orthogonality, meaning no similarity.
|
||||
It's a simple yet effective measure for text-based retrieval, as it considers the orientation but not the magnitude of vectors.
|
||||
|
||||
Dot Product
|
||||
-----------
|
||||
|
||||
Coordinate definition:
|
||||
.. math::
|
||||
|
||||
A\cdot B = \sum_{i=1}^{i=n}A_i B_i
|
||||
|
||||
Geometric definition:
|
||||
.. math::
|
||||
|
||||
A\cdot B = \|A\|\|B\|\cos(\theta)
|
||||
|
||||
The dot product between two vectors provides a measure of how similar the vectors are in terms of direction and magnitude.
|
||||
In information retrieval, the dot product is often used in vector space models, particularly when dealing with pre-trained word or sentence embeddings.
|
||||
A higher dot product indicates that the query and document are closely aligned in the vector space.
|
||||
|
||||
Reference in New Issue
Block a user