wehub/flagopen--flagembedding
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 13:39:21 +08:00
commit bc34f6df14
1149 changed files with 328099 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/source/tutorial/4_Evaluation/4.1.1.ipynb
+509
View File
@@ -0,0 +1,509 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Evaluation"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Evaluation is a crucial part in all machine learning tasks. In this notebook, we will walk through the whole pipeline of evaluating the performance of an embedding model on [MS Marco](https://microsoft.github.io/msmarco/), and use three metrics to show its performance."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 0: Setup"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Install the dependencies in the environment."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%pip install -U FlagEmbedding faiss-cpu"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 1: Load Dataset"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"First, download the queries and MS Marco from Huggingface Dataset"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"from datasets import load_dataset\n",
"import numpy as np\n",
"\n",
"data = load_dataset(\"namespace-Pt/msmarco\", split=\"dev\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Considering time cost, we will use the truncated dataset in this tutorial. `queries` contains the first 100 queries from the dataset. `corpus` is formed by the positives of the the first 5,000 queries."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"queries = np.array(data[:100][\"query\"])\n",
"corpus = sum(data[:5000][\"positive\"], [])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If you have GPU and would like to try out the full evaluation of MS Marco, uncomment and run the following cell:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# data = load_dataset(\"namespace-Pt/msmarco\", split=\"dev\")\n",
"# queries = np.array(data[\"query\"])\n",
"\n",
"# corpus = load_dataset(\"namespace-PT/msmarco-corpus\", split=\"train\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 2: Embedding"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Choose the embedding model that we would like to evaluate, and encode the corpus to embeddings."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Inference Embeddings: 100%|██████████| 21/21 [02:10<00:00, 6.22s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"shape of the corpus embeddings: (5331, 768)\n",
"data type of the embeddings: float32\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\n"
]
}
],
"source": [
"from FlagEmbedding import FlagModel\n",
"\n",
"# get the BGE embedding model\n",
"model = FlagModel('BAAI/bge-base-en-v1.5',\n",
" query_instruction_for_retrieval=\"Represent this sentence for searching relevant passages:\",\n",
" use_fp16=True)\n",
"\n",
"# get the embedding of the corpus\n",
"corpus_embeddings = model.encode(corpus)\n",
"\n",
"print(\"shape of the corpus embeddings:\", corpus_embeddings.shape)\n",
"print(\"data type of the embeddings: \", corpus_embeddings.dtype)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 3: Indexing"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We use the index_factory() functions to create a Faiss index we want:\n",
"\n",
"- The first argument `dim` is the dimension of the vector space, in this case is 768 if you're using bge-base-en-v1.5.\n",
"\n",
"- The second argument `'Flat'` makes the index do exhaustive search.\n",
"\n",
"- The thrid argument `faiss.METRIC_INNER_PRODUCT` tells the index to use inner product as the distance metric."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"total number of vectors: 5331\n"
]
}
],
"source": [
"import faiss\n",
"\n",
"# get the length of our embedding vectors, vectors by bge-base-en-v1.5 have length 768\n",
"dim = corpus_embeddings.shape[-1]\n",
"\n",
"# create the faiss index and store the corpus embeddings into the vector space\n",
"index = faiss.index_factory(dim, 'Flat', faiss.METRIC_INNER_PRODUCT)\n",
"corpus_embeddings = corpus_embeddings.astype(np.float32)\n",
"# train and add the embeddings to the index\n",
"index.train(corpus_embeddings)\n",
"index.add(corpus_embeddings)\n",
"\n",
"print(f\"total number of vectors: {index.ntotal}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Since the embedding process is time consuming, it's a good choice to save the index for reproduction or other experiments.\n",
"\n",
"Uncomment the following lines to save the index."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
"# path = \"./index.bin\"\n",
"# faiss.write_index(index, path)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If you already have stored index in your local directory, you can load it by:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# index = faiss.read_index(\"./index.bin\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 4: Retrieval"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Get the embeddings of all the queries, and get their corresponding ground truth answers for evaluation."
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [],
"source": [
"query_embeddings = model.encode_queries(queries)\n",
"ground_truths = [d[\"positive\"] for d in data]\n",
"corpus = np.asarray(corpus)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Use the faiss index to search top $k$ answers of each query."
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Searching: 100%|██████████| 1/1 [00:00<00:00, 20.91it/s]\n"
]
}
],
"source": [
"from tqdm import tqdm\n",
"\n",
"res_scores, res_ids, res_text = [], [], []\n",
"query_size = len(query_embeddings)\n",
"batch_size = 256\n",
"# The cutoffs we will use during evaluation, and set k to be the maximum of the cutoffs.\n",
"cut_offs = [1, 10]\n",
"k = max(cut_offs)\n",
"\n",
"for i in tqdm(range(0, query_size, batch_size), desc=\"Searching\"):\n",
" q_embedding = query_embeddings[i: min(i+batch_size, query_size)].astype(np.float32)\n",
" # search the top k answers for each of the queries\n",
" score, idx = index.search(q_embedding, k=k)\n",
" res_scores += list(score)\n",
" res_ids += list(idx)\n",
" res_text += list(corpus[idx])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 5: Evaluate"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 5.1 Recall"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Recall represents the model's capability of correctly predicting positive instances from all the actual positive samples in the dataset.\n",
"\n",
"$$\\textbf{Recall}=\\frac{\\text{True Positives}}{\\text{True Positives}+\\text{False Negatives}}$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Recall is useful when the cost of false negatives is high. In other words, we are trying to find all objects of the positive class, even if this results in some false positives. This attribute makes recall a useful metric for text retrieval tasks."
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"recall@1: 0.97\n",
"recall@10: 1.0\n"
]
}
],
"source": [
"def calc_recall(preds, truths, cutoffs):\n",
" recalls = np.zeros(len(cutoffs))\n",
" for text, truth in zip(preds, truths):\n",
" for i, c in enumerate(cutoffs):\n",
" recall = np.intersect1d(truth, text[:c])\n",
" recalls[i] += len(recall) / max(min(c, len(truth)), 1)\n",
" recalls /= len(preds)\n",
" return recalls\n",
"\n",
"recalls = calc_recall(res_text, ground_truths, cut_offs)\n",
"for i, c in enumerate(cut_offs):\n",
" print(f\"recall@{c}: {recalls[i]}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 5.2 MRR"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Mean Reciprocal Rank ([MRR](https://en.wikipedia.org/wiki/Mean_reciprocal_rank)) is a widely used metric in information retrieval to evaluate the effectiveness of a system. It measures the rank position of the first relevant result in a list of search results.\n",
"\n",
"$$MRR=\\frac{1}{|Q|}\\sum_{i=1}^{|Q|}\\frac{1}{rank_i}$$\n",
"\n",
"where \n",
"- $|Q|$ is the total number of queries.\n",
"- $rank_i$ is the rank position of the first relevant document of the i-th query."
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [],
"source": [
"def MRR(preds, truth, cutoffs):\n",
" mrr = [0 for _ in range(len(cutoffs))]\n",
" for pred, t in zip(preds, truth):\n",
" for i, c in enumerate(cutoffs):\n",
" for j, p in enumerate(pred):\n",
" if j < c and p in t:\n",
" mrr[i] += 1/(j+1)\n",
" break\n",
" mrr = [k/len(preds) for k in mrr]\n",
" return mrr"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"MRR@1: 0.97\n",
"MRR@10: 0.9825\n"
]
}
],
"source": [
"mrr = MRR(res_text, ground_truths, cut_offs)\n",
"for i, c in enumerate(cut_offs):\n",
" print(f\"MRR@{c}: {mrr[i]}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 5.3 nDCG"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Normalized Discounted cumulative gain (nDCG) measures the quality of a ranked list of search results by considering both the position of the relevant documents and their graded relevance scores. The calculation of nDCG involves two main steps:\n",
"\n",
"1. Discounted cumulative gain (DCG) measures the ranking quality in retrieval tasks.\n",
"\n",
"$$DCG_p=\\sum_{i=1}^p\\frac{2^{rel_i}-1}{\\log_2(i+1)}$$\n",
"\n",
"2. Normalized by ideal DCG to make it comparable across queries.\n",
"$$nDCG_p=\\frac{DCG_p}{IDCG_p}$$\n",
"where $IDCG$ is the maximum possible DCG for a given set of documents, assuming they are perfectly ranked in order of relevance."
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [],
"source": [
"pred_hard_encodings = []\n",
"for pred, label in zip(res_text, ground_truths):\n",
" pred_hard_encoding = list(np.isin(pred, label).astype(int))\n",
" pred_hard_encodings.append(pred_hard_encoding)"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"nDCG@1: 0.97\n",
"nDCG@10: 0.9869253606521631\n"
]
}
],
"source": [
"from sklearn.metrics import ndcg_score\n",
"\n",
"for i, c in enumerate(cut_offs):\n",
" nDCG = ndcg_score(pred_hard_encodings, res_scores, k=c)\n",
" print(f\"nDCG@{c}: {nDCG}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Congrats! You have walked through a full pipeline of evaluating an embedding model. Feel free to play with different datasets and models!"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "base",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.13"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
docs/source/tutorial/4_Evaluation/4.2.1.ipynb
+436
View File
@@ -0,0 +1,436 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# MTEB"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"For evaluation of embedding models, MTEB is one of the most well-known benchmark. In this tutorial, we'll introduce MTEB, its basic usage, and evaluate how your model performs on the MTEB leaderboard."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 0. Installation"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Install the packages we will use in your environment:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%%capture\n",
"%pip install sentence_transformers mteb"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. Intro"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The [Massive Text Embedding Benchmark (MTEB)](https://github.com/embeddings-benchmark/mteb) is a large-scale evaluation framework designed to assess the performance of text embedding models across a wide variety of natural language processing (NLP) tasks. Introduced to standardize and improve the evaluation of text embeddings, MTEB is crucial for assessing how well these models generalize across various real-world applications. It contains a wide range of datasets in eight main NLP tasks and different languages, and provides an easy pipeline for evaluation.\n",
"\n",
"MTEB is also well known for the MTEB leaderboard, which contains a ranking of the latest first-class embedding models. We'll cover that in the next tutorial. Now let's have a look on how to use MTEB to do evaluation easily."
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [],
"source": [
"import mteb\n",
"from sentence_transformers import SentenceTransformer"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now let's take a look at how to use MTEB to do a quick evaluation.\n",
"\n",
"First we load the model that we would like to evaluate on:"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [],
"source": [
"model_name = \"BAAI/bge-base-en-v1.5\"\n",
"model = SentenceTransformer(model_name)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Below is the list of datasets of retrieval used by MTEB's English leaderboard.\n",
"\n",
"MTEB directly use the open source benchmark BEIR in its retrieval part, which contains 15 datasets (note there are 12 subsets of CQADupstack)."
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [],
"source": [
"retrieval_tasks = [\n",
" \"ArguAna\",\n",
" \"ClimateFEVER\",\n",
" \"CQADupstackAndroidRetrieval\",\n",
" \"CQADupstackEnglishRetrieval\",\n",
" \"CQADupstackGamingRetrieval\",\n",
" \"CQADupstackGisRetrieval\",\n",
" \"CQADupstackMathematicaRetrieval\",\n",
" \"CQADupstackPhysicsRetrieval\",\n",
" \"CQADupstackProgrammersRetrieval\",\n",
" \"CQADupstackStatsRetrieval\",\n",
" \"CQADupstackTexRetrieval\",\n",
" \"CQADupstackUnixRetrieval\",\n",
" \"CQADupstackWebmastersRetrieval\",\n",
" \"CQADupstackWordpressRetrieval\",\n",
" \"DBPedia\",\n",
" \"FEVER\",\n",
" \"FiQA2018\",\n",
" \"HotpotQA\",\n",
" \"MSMARCO\",\n",
" \"NFCorpus\",\n",
" \"NQ\",\n",
" \"QuoraRetrieval\",\n",
" \"SCIDOCS\",\n",
" \"SciFact\",\n",
" \"Touche2020\",\n",
" \"TRECCOVID\",\n",
"]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"For demonstration, let's just run the first one, \"ArguAna\".\n",
"\n",
"For a full list of tasks and languages that MTEB supports, check the [page](https://github.com/embeddings-benchmark/mteb/blob/18662380f0f476db3d170d0926892045aa9f74ee/docs/tasks.md)."
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [],
"source": [
"tasks = mteb.get_tasks(tasks=retrieval_tasks[:1])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Then, create and initialize an MTEB instance with our chosen tasks, and run the evaluation process."
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #262626; text-decoration-color: #262626\">───────────────────────────────────────────────── </span><span style=\"font-weight: bold\">Selected tasks </span><span style=\"color: #262626; text-decoration-color: #262626\"> ─────────────────────────────────────────────────</span>\n",
"</pre>\n"
],
"text/plain": [
"\u001b[38;5;235m───────────────────────────────────────────────── \u001b[0m\u001b[1mSelected tasks \u001b[0m\u001b[38;5;235m ─────────────────────────────────────────────────\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Retrieval</span>\n",
"</pre>\n"
],
"text/plain": [
"\u001b[1mRetrieval\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"> - ArguAna, <span style=\"color: #626262; text-decoration-color: #626262; font-style: italic\">s2p</span>\n",
"</pre>\n"
],
"text/plain": [
" - ArguAna, \u001b[3;38;5;241ms2p\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
"\n",
"</pre>\n"
],
"text/plain": [
"\n",
"\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Batches: 100%|██████████| 44/44 [00:41<00:00, 1.06it/s]\n",
"Batches: 100%|██████████| 272/272 [03:36<00:00, 1.26it/s]\n"
]
}
],
"source": [
"# use the tasks we chose to initialize the MTEB instance\n",
"evaluation = mteb.MTEB(tasks=tasks)\n",
"\n",
"# call run() with the model and output_folder\n",
"results = evaluation.run(model, output_folder=\"results\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The results should be stored in `{output_folder}/{model_name}/{model_revision}/{task_name}.json`.\n",
"\n",
"Openning the json file you should see contents as below, which are the evaluation results on \"ArguAna\" with different metrics on cutoffs from 1 to 1000."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```python\n",
"{\n",
" \"dataset_revision\": \"c22ab2a51041ffd869aaddef7af8d8215647e41a\",\n",
" \"evaluation_time\": 260.14976954460144,\n",
" \"kg_co2_emissions\": null,\n",
" \"mteb_version\": \"1.14.17\",\n",
" \"scores\": {\n",
" \"test\": [\n",
" {\n",
" \"hf_subset\": \"default\",\n",
" \"languages\": [\n",
" \"eng-Latn\"\n",
" ],\n",
" \"main_score\": 0.63616,\n",
" \"map_at_1\": 0.40754,\n",
" \"map_at_10\": 0.55773,\n",
" \"map_at_100\": 0.56344,\n",
" \"map_at_1000\": 0.56347,\n",
" \"map_at_20\": 0.56202,\n",
" \"map_at_3\": 0.51932,\n",
" \"map_at_5\": 0.54023,\n",
" \"mrr_at_1\": 0.4139402560455192,\n",
" \"mrr_at_10\": 0.5603739077423295,\n",
" \"mrr_at_100\": 0.5660817425350153,\n",
" \"mrr_at_1000\": 0.5661121884705748,\n",
" \"mrr_at_20\": 0.564661930998293,\n",
" \"mrr_at_3\": 0.5208629682313899,\n",
" \"mrr_at_5\": 0.5429113323850182,\n",
" \"nauc_map_at_1000_diff1\": 0.15930478114759905,\n",
" \"nauc_map_at_1000_max\": -0.06396189194646361,\n",
" \"nauc_map_at_1000_std\": -0.13168797291549253,\n",
" \"nauc_map_at_100_diff1\": 0.15934819555197366,\n",
" \"nauc_map_at_100_max\": -0.06389635013430676,\n",
" \"nauc_map_at_100_std\": -0.13164524259533786,\n",
" \"nauc_map_at_10_diff1\": 0.16057318234658585,\n",
" \"nauc_map_at_10_max\": -0.060962623117325254,\n",
" \"nauc_map_at_10_std\": -0.1300413865104607,\n",
" \"nauc_map_at_1_diff1\": 0.17346152653542332,\n",
" \"nauc_map_at_1_max\": -0.09705499215630589,\n",
" \"nauc_map_at_1_std\": -0.14726476953035533,\n",
" \"nauc_map_at_20_diff1\": 0.15956349246366208,\n",
" \"nauc_map_at_20_max\": -0.06259296677860492,\n",
" \"nauc_map_at_20_std\": -0.13097093150054095,\n",
" \"nauc_map_at_3_diff1\": 0.15620049317363813,\n",
" \"nauc_map_at_3_max\": -0.06690213479396273,\n",
" \"nauc_map_at_3_std\": -0.13440904793529648,\n",
" \"nauc_map_at_5_diff1\": 0.1557795701081579,\n",
" \"nauc_map_at_5_max\": -0.06255283252590663,\n",
" \"nauc_map_at_5_std\": -0.1355361594910923,\n",
" \"nauc_mrr_at_1000_diff1\": 0.1378988612808882,\n",
" \"nauc_mrr_at_1000_max\": -0.07507962333910836,\n",
" \"nauc_mrr_at_1000_std\": -0.12969109830101241,\n",
" \"nauc_mrr_at_100_diff1\": 0.13794450668758515,\n",
" \"nauc_mrr_at_100_max\": -0.07501290390362861,\n",
" \"nauc_mrr_at_100_std\": -0.12964855554504057,\n",
" \"nauc_mrr_at_10_diff1\": 0.1396047981645623,\n",
" \"nauc_mrr_at_10_max\": -0.07185174301688693,\n",
" \"nauc_mrr_at_10_std\": -0.12807325096717753,\n",
" \"nauc_mrr_at_1_diff1\": 0.15610387932529113,\n",
" \"nauc_mrr_at_1_max\": -0.09824591983546396,\n",
" \"nauc_mrr_at_1_std\": -0.13914318784294258,\n",
" \"nauc_mrr_at_20_diff1\": 0.1382786098284509,\n",
" \"nauc_mrr_at_20_max\": -0.07364476417961506,\n",
" \"nauc_mrr_at_20_std\": -0.12898192060943495,\n",
" \"nauc_mrr_at_3_diff1\": 0.13118224861025093,\n",
" \"nauc_mrr_at_3_max\": -0.08164985279853691,\n",
" \"nauc_mrr_at_3_std\": -0.13241573571401533,\n",
" \"nauc_mrr_at_5_diff1\": 0.1346130730317385,\n",
" \"nauc_mrr_at_5_max\": -0.07404093236468848,\n",
" \"nauc_mrr_at_5_std\": -0.1340775377068567,\n",
" \"nauc_ndcg_at_1000_diff1\": 0.15919987960292029,\n",
" \"nauc_ndcg_at_1000_max\": -0.05457945565481172,\n",
" \"nauc_ndcg_at_1000_std\": -0.12457339152558143,\n",
" \"nauc_ndcg_at_100_diff1\": 0.1604091882521101,\n",
" \"nauc_ndcg_at_100_max\": -0.05281549383775287,\n",
" \"nauc_ndcg_at_100_std\": -0.12347288098914058,\n",
" \"nauc_ndcg_at_10_diff1\": 0.1657018523692905,\n",
" \"nauc_ndcg_at_10_max\": -0.036222943297402846,\n",
" \"nauc_ndcg_at_10_std\": -0.11284619565817842,\n",
" \"nauc_ndcg_at_1_diff1\": 0.17346152653542332,\n",
" \"nauc_ndcg_at_1_max\": -0.09705499215630589,\n",
" \"nauc_ndcg_at_1_std\": -0.14726476953035533,\n",
" \"nauc_ndcg_at_20_diff1\": 0.16231721725673165,\n",
" \"nauc_ndcg_at_20_max\": -0.04147115653921931,\n",
" \"nauc_ndcg_at_20_std\": -0.11598700704312062,\n",
" \"nauc_ndcg_at_3_diff1\": 0.15256475371124711,\n",
" \"nauc_ndcg_at_3_max\": -0.05432154580979357,\n",
" \"nauc_ndcg_at_3_std\": -0.12841084787822227,\n",
" \"nauc_ndcg_at_5_diff1\": 0.15236205846534961,\n",
" \"nauc_ndcg_at_5_max\": -0.04356123278888682,\n",
" \"nauc_ndcg_at_5_std\": -0.12942556865700913,\n",
" \"nauc_precision_at_1000_diff1\": -0.038790629929866066,\n",
" \"nauc_precision_at_1000_max\": 0.3630826341915611,\n",
" \"nauc_precision_at_1000_std\": 0.4772189839676386,\n",
" \"nauc_precision_at_100_diff1\": 0.32118609204433185,\n",
" \"nauc_precision_at_100_max\": 0.4740132817600036,\n",
" \"nauc_precision_at_100_std\": 0.3456396169952022,\n",
" \"nauc_precision_at_10_diff1\": 0.22279659689895104,\n",
" \"nauc_precision_at_10_max\": 0.16823918613191954,\n",
" \"nauc_precision_at_10_std\": 0.0377209694331257,\n",
" \"nauc_precision_at_1_diff1\": 0.17346152653542332,\n",
" \"nauc_precision_at_1_max\": -0.09705499215630589,\n",
" \"nauc_precision_at_1_std\": -0.14726476953035533,\n",
" \"nauc_precision_at_20_diff1\": 0.23025740175221762,\n",
" \"nauc_precision_at_20_max\": 0.2892313928157665,\n",
" \"nauc_precision_at_20_std\": 0.13522755012490692,\n",
" \"nauc_precision_at_3_diff1\": 0.1410889527057097,\n",
" \"nauc_precision_at_3_max\": -0.010771302313530132,\n",
" \"nauc_precision_at_3_std\": -0.10744937823276193,\n",
" \"nauc_precision_at_5_diff1\": 0.14012953903010988,\n",
" \"nauc_precision_at_5_max\": 0.03977485677045894,\n",
" \"nauc_precision_at_5_std\": -0.10292184602358977,\n",
" \"nauc_recall_at_1000_diff1\": -0.03879062992990034,\n",
" \"nauc_recall_at_1000_max\": 0.36308263419153386,\n",
" \"nauc_recall_at_1000_std\": 0.47721898396760526,\n",
" \"nauc_recall_at_100_diff1\": 0.3211860920443005,\n",
" \"nauc_recall_at_100_max\": 0.4740132817599919,\n",
" \"nauc_recall_at_100_std\": 0.345639616995194,\n",
" \"nauc_recall_at_10_diff1\": 0.22279659689895054,\n",
" \"nauc_recall_at_10_max\": 0.16823918613192046,\n",
" \"nauc_recall_at_10_std\": 0.037720969433127145,\n",
" \"nauc_recall_at_1_diff1\": 0.17346152653542332,\n",
" \"nauc_recall_at_1_max\": -0.09705499215630589,\n",
" \"nauc_recall_at_1_std\": -0.14726476953035533,\n",
" \"nauc_recall_at_20_diff1\": 0.23025740175221865,\n",
" \"nauc_recall_at_20_max\": 0.2892313928157675,\n",
" \"nauc_recall_at_20_std\": 0.13522755012490456,\n",
" \"nauc_recall_at_3_diff1\": 0.14108895270570979,\n",
" \"nauc_recall_at_3_max\": -0.010771302313529425,\n",
" \"nauc_recall_at_3_std\": -0.10744937823276134,\n",
" \"nauc_recall_at_5_diff1\": 0.14012953903010958,\n",
" \"nauc_recall_at_5_max\": 0.039774856770459645,\n",
" \"nauc_recall_at_5_std\": -0.10292184602358935,\n",
" \"ndcg_at_1\": 0.40754,\n",
" \"ndcg_at_10\": 0.63616,\n",
" \"ndcg_at_100\": 0.66063,\n",
" \"ndcg_at_1000\": 0.6613,\n",
" \"ndcg_at_20\": 0.65131,\n",
" \"ndcg_at_3\": 0.55717,\n",
" \"ndcg_at_5\": 0.59461,\n",
" \"precision_at_1\": 0.40754,\n",
" \"precision_at_10\": 0.08841,\n",
" \"precision_at_100\": 0.00991,\n",
" \"precision_at_1000\": 0.001,\n",
" \"precision_at_20\": 0.04716,\n",
" \"precision_at_3\": 0.22238,\n",
" \"precision_at_5\": 0.15149,\n",
" \"recall_at_1\": 0.40754,\n",
" \"recall_at_10\": 0.88407,\n",
" \"recall_at_100\": 0.99147,\n",
" \"recall_at_1000\": 0.99644,\n",
" \"recall_at_20\": 0.9431,\n",
" \"recall_at_3\": 0.66714,\n",
" \"recall_at_5\": 0.75747\n",
" }\n",
" ]\n",
" },\n",
" \"task_name\": \"ArguAna\"\n",
"}\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we've successfully run the evaluation using mteb! In the next tutorial, we'll show how to evaluate your model on the whole 56 tasks of English MTEB and compete with models on the leaderboard."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "base",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.13"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
docs/source/tutorial/4_Evaluation/4.2.2.ipynb
+302
View File
@@ -0,0 +1,302 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# MTEB Leaderboard"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In the last tutorial we show how to evaluate an embedding model on an dataset supported by MTEB. In this tutorial, we will go through how to do a full evaluation and compare the results with MTEB English leaderboard.\n",
"\n",
"Caution: Evaluation on the full Eng MTEB is very time consuming even with GPU. So we encourage you to go through the notebook to have an idea. And run the experiment when you have enough computing resource and time."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 0. Installation"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Install the packages we will use in your environment:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%%capture\n",
"%pip install sentence_transformers mteb"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. Run the Evaluation"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The MTEB English leaderboard contains 56 datasets on 7 tasks:\n",
"1. **Classification**: Use the embeddings to train a logistic regression on the train set and is scored on the test set. F1 is the main metric.\n",
"2. **Clustering**: Train a mini-batch k-means model with batch size 32 and k equals to the number of different labels. Then score using v-measure.\n",
"3. **Pair Classification**: A pair of text inputs is provided and a label which is a binary variable needs to be assigned. The main metric is average precision score.\n",
"4. **Reranking**: Rank a list of relevant and irrelevant reference texts according to a query. Metrics are mean MRR@k and MAP.\n",
"5. **Retrieval**: Each dataset comprises corpus, queries, and a mapping that links each query to its relevant documents within the corpus. The goal is to retrieve relevant documents for each query. The main metric is nDCG@k. MTEB directly adopts BEIR for the retrieval task.\n",
"6. **Semantic Textual Similarity (STS)**: Determine the similarity between each sentence pair. Spearman correlation based on cosine\n",
"similarity serves as the main metric.\n",
"7. **Summarization**: Only 1 dataset is used in this task. Score the machine-generated summaries to human-written summaries by computing distances of their embeddings. The main metric is also Spearman correlation based on cosine similarity.\n",
"\n",
"The benchmark is widely accepted by researchers and engineers to fairly evaluate and compare the performance of the models they train. Now let's take a look at the whole evaluation pipeline"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Import the `MTEB_MAIN_EN` to check the all 56 datasets."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"['AmazonCounterfactualClassification', 'AmazonPolarityClassification', 'AmazonReviewsClassification', 'ArguAna', 'ArxivClusteringP2P', 'ArxivClusteringS2S', 'AskUbuntuDupQuestions', 'BIOSSES', 'Banking77Classification', 'BiorxivClusteringP2P', 'BiorxivClusteringS2S', 'CQADupstackAndroidRetrieval', 'CQADupstackEnglishRetrieval', 'CQADupstackGamingRetrieval', 'CQADupstackGisRetrieval', 'CQADupstackMathematicaRetrieval', 'CQADupstackPhysicsRetrieval', 'CQADupstackProgrammersRetrieval', 'CQADupstackStatsRetrieval', 'CQADupstackTexRetrieval', 'CQADupstackUnixRetrieval', 'CQADupstackWebmastersRetrieval', 'CQADupstackWordpressRetrieval', 'ClimateFEVER', 'DBPedia', 'EmotionClassification', 'FEVER', 'FiQA2018', 'HotpotQA', 'ImdbClassification', 'MSMARCO', 'MTOPDomainClassification', 'MTOPIntentClassification', 'MassiveIntentClassification', 'MassiveScenarioClassification', 'MedrxivClusteringP2P', 'MedrxivClusteringS2S', 'MindSmallReranking', 'NFCorpus', 'NQ', 'QuoraRetrieval', 'RedditClustering', 'RedditClusteringP2P', 'SCIDOCS', 'SICK-R', 'STS12', 'STS13', 'STS14', 'STS15', 'STS16', 'STS17', 'STS22', 'STSBenchmark', 'SciDocsRR', 'SciFact', 'SprintDuplicateQuestions', 'StackExchangeClustering', 'StackExchangeClusteringP2P', 'StackOverflowDupQuestions', 'SummEval', 'TRECCOVID', 'Touche2020', 'ToxicConversationsClassification', 'TweetSentimentExtractionClassification', 'TwentyNewsgroupsClustering', 'TwitterSemEval2015', 'TwitterURLCorpus']\n"
]
}
],
"source": [
"import mteb\n",
"from mteb.benchmarks import MTEB_MAIN_EN\n",
"\n",
"print(MTEB_MAIN_EN.tasks)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Load the model we want to evaluate:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from sentence_transformers import SentenceTransformer\n",
"\n",
"model_name = \"BAAI/bge-base-en-v1.5\"\n",
"model = SentenceTransformer(model_name)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Alternatively, MTEB provides popular models on their leaderboard in order to reproduce their results."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model_name = \"BAAI/bge-base-en-v1.5\"\n",
"model = mteb.get_model(model_name)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Then start to evaluate on each dataset:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"for task in MTEB_MAIN_EN.tasks:\n",
" # get the test set to evaluate on\n",
" eval_splits = [\"dev\"] if task == \"MSMARCO\" else [\"test\"]\n",
" evaluation = mteb.MTEB(\n",
" tasks=[task], task_langs=[\"en\"]\n",
" ) # Remove \"en\" to run all available languages\n",
" evaluation.run(\n",
" model, output_folder=\"results\", eval_splits=eval_splits\n",
" )"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2. Submit to MTEB Leaderboard"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"After the evaluation is done, all the evaluation results should be stored in `results/{model_name}/{model_revision}`.\n",
"\n",
"Then run the following shell command to create the model_card.md. Change {model_name} and {model_revision} to your path."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!mteb create_meta --results_folder results/{model_name}/{model_revision} --output_path model_card.md"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"For the case that the readme of that model already exists:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# !mteb create_meta --results_folder results/{model_name}/{model_revision} --output_path model_card.md --from_existing your_existing_readme.md "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Copy and paste the contents of model_card.md to the top of README.md of your model on HF Hub. Now relax and wait for the daily refresh of leaderboard. Your model will show up soon!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3. Partially Evaluate"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Note that you don't need to finish all the tasks to get on to the leaderboard.\n",
"\n",
"For example you fine-tune a model's ability on clustering. And you only care about how your model performs with respoect to clustering, but not the other tasks. Then you can just test its performance on the clustering tasks of MTEB and submit to the leaderboard."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"TASK_LIST_CLUSTERING = [\n",
" \"ArxivClusteringP2P\",\n",
" \"ArxivClusteringS2S\",\n",
" \"BiorxivClusteringP2P\",\n",
" \"BiorxivClusteringS2S\",\n",
" \"MedrxivClusteringP2P\",\n",
" \"MedrxivClusteringS2S\",\n",
" \"RedditClustering\",\n",
" \"RedditClusteringP2P\",\n",
" \"StackExchangeClustering\",\n",
" \"StackExchangeClusteringP2P\",\n",
" \"TwentyNewsgroupsClustering\",\n",
"]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Run the evaluation with only clustering tasks:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"evaluation = mteb.MTEB(tasks=TASK_LIST_CLUSTERING)\n",
"\n",
"results = evaluation.run(model, output_folder=\"results\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Then repeat Step 2 to submit your model. After the leaderboard refresh, you can find your model in the \"Clustering\" section of the leaderboard."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 4. Future Work"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"MTEB is working on a new version of English benchmark. It contains updated and concise tasks and will make the evaluation process faster.\n",
"\n",
"Please check out their [GitHub](https://github.com/embeddings-benchmark/mteb) page for future updates and releases."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "base",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.13"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
docs/source/tutorial/4_Evaluation/4.2.3.ipynb
+240
View File
@@ -0,0 +1,240 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# C-MTEB"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"C-MTEB is the largest benchmark for Chinese text embeddings, similar to MTEB. In this tutorial, we will go through how to evaluate an embedding model's ability on Chinese tasks in C-MTEB."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 0. Installation"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"First install dependent packages:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%pip install FlagEmbedding mteb"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. Datasets"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"C-MTEB uses similar task splits and metrics as English MTEB. It contains 35 datasets in 6 different tasks: Classification, Clustering, Pair Classification, Reranking, Retrieval, and Semantic Textual Similarity (STS). \n",
"\n",
"1. **Classification**: Use the embeddings to train a logistic regression on the train set and is scored on the test set. F1 is the main metric.\n",
"2. **Clustering**: Train a mini-batch k-means model with batch size 32 and k equals to the number of different labels. Then score using v-measure.\n",
"3. **Pair Classification**: A pair of text inputs is provided and a label which is a binary variable needs to be assigned. The main metric is average precision score.\n",
"4. **Reranking**: Rank a list of relevant and irrelevant reference texts according to a query. Metrics are mean MRR@k and MAP.\n",
"5. **Retrieval**: Each dataset comprises corpus, queries, and a mapping that links each query to its relevant documents within the corpus. The goal is to retrieve relevant documents for each query. The main metric is nDCG@k. MTEB directly adopts BEIR for the retrieval task.\n",
"6. **Semantic Textual Similarity (STS)**: Determine the similarity between each sentence pair. Spearman correlation based on cosine\n",
"similarity serves as the main metric.\n",
"\n",
"\n",
"Check the [HF page](https://huggingface.co/C-MTEB) for the details of each dataset."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"ChineseTaskList = [\n",
" 'TNews', 'IFlyTek', 'MultilingualSentiment', 'JDReview', 'OnlineShopping', 'Waimai',\n",
" 'CLSClusteringS2S.v2', 'CLSClusteringP2P.v2', 'ThuNewsClusteringS2S.v2', 'ThuNewsClusteringP2P.v2',\n",
" 'Ocnli', 'Cmnli',\n",
" 'T2Reranking', 'MMarcoReranking', 'CMedQAv1-reranking', 'CMedQAv2-reranking',\n",
" 'T2Retrieval', 'MMarcoRetrieval', 'DuRetrieval', 'CovidRetrieval', 'CmedqaRetrieval', 'EcomRetrieval', 'MedicalRetrieval', 'VideoRetrieval',\n",
" 'ATEC', 'BQ', 'LCQMC', 'PAWSX', 'STSB', 'AFQMC', 'QBQTC'\n",
"]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2. Model"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"First, load the model for evaluation. Note that the instruction here is used for retreival tasks."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from ...C_MTEB.flag_dres_model import FlagDRESModel\n",
"\n",
"instruction = \"为这个句子生成表示以用于检索相关文章：\"\n",
"model_name = \"BAAI/bge-base-zh-v1.5\"\n",
"\n",
"model = FlagDRESModel(model_name_or_path=\"BAAI/bge-base-zh-v1.5\",\n",
" query_instruction_for_retrieval=instruction,\n",
" pooling_method=\"cls\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Otherwise, you can load a model using sentence_transformers:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from sentence_transformers import SentenceTransformer\n",
"\n",
"model = SentenceTransformer(\"PATH_TO_MODEL\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Or implement a class following the structure below:\n",
"\n",
"```python\n",
"class MyModel():\n",
" def __init__(self):\n",
" \"\"\"initialize the tokenizer and model\"\"\"\n",
" pass\n",
"\n",
" def encode(self, sentences, batch_size=32, **kwargs):\n",
" \"\"\" Returns a list of embeddings for the given sentences.\n",
" Args:\n",
" sentences (`List[str]`): List of sentences to encode\n",
" batch_size (`int`): Batch size for the encoding\n",
"\n",
" Returns:\n",
" `List[np.ndarray]` or `List[tensor]`: List of embeddings for the given sentences\n",
" \"\"\"\n",
" pass\n",
"\n",
"model = MyModel()\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3. Evaluate"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"After we've prepared the dataset and model, we can start the evaluation. For time efficiency, we highly recommend to use GPU for evaluation."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import mteb\n",
"from mteb import MTEB\n",
"\n",
"tasks = mteb.get_tasks(ChineseTaskList)\n",
"\n",
"for task in tasks:\n",
" evaluation = MTEB(tasks=[task])\n",
" evaluation.run(model, output_folder=f\"zh_results/{model_name.split('/')[-1]}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 4. Submit to MTEB Leaderboard"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"After the evaluation is done, all the evaluation results should be stored in `zh_results/{model_name}/`.\n",
"\n",
"Then run the following shell command to create the model_card.md. Change {model_name} and its following to your path."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!!mteb create_meta --results_folder results/{model_name}/ --output_path model_card.md"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Copy and paste the contents of model_card.md to the top of README.md of your model on HF Hub. Then goto the [MTEB leaderboard](https://huggingface.co/spaces/mteb/leaderboard) and choose the Chinese leaderboard to find your model! It will appear soon after the website's daily refresh."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.2"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
docs/source/tutorial/4_Evaluation/4.3.1.ipynb
+152
View File
@@ -0,0 +1,152 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Evaluation Using Sentence Transformers"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In this tutorial, we will go through how to use the Sentence Tranformers library to do evaluation."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 0. Installation"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%pip install -U sentence-transformers"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from sentence_transformers import SentenceTransformer\n",
"\n",
"# Load a model\n",
"model = SentenceTransformer('all-MiniLM-L6-v2')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. Retrieval"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's choose retrieval as the first task"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import random\n",
"\n",
"from sentence_transformers.evaluation import InformationRetrievalEvaluator\n",
"\n",
"from datasets import load_dataset"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"BeIR is a well known benchmark for retrieval. Let's use the xxx dataset for our evaluation."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Load the Quora IR dataset (https://huggingface.co/datasets/BeIR/quora, https://huggingface.co/datasets/BeIR/quora-qrels)\n",
"corpus = load_dataset(\"BeIR/quora\", \"corpus\", split=\"corpus\")\n",
"queries = load_dataset(\"BeIR/quora\", \"queries\", split=\"queries\")\n",
"relevant_docs_data = load_dataset(\"BeIR/quora-qrels\", split=\"validation\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Shrink the corpus size heavily to only the relevant documents + 10,000 random documents\n",
"required_corpus_ids = list(map(str, relevant_docs_data[\"corpus-id\"]))\n",
"required_corpus_ids += random.sample(corpus[\"_id\"], k=10_000)\n",
"corpus = corpus.filter(lambda x: x[\"_id\"] in required_corpus_ids)\n",
"\n",
"# Convert the datasets to dictionaries\n",
"corpus = dict(zip(corpus[\"_id\"], corpus[\"text\"])) # Our corpus (cid => document)\n",
"queries = dict(zip(queries[\"_id\"], queries[\"text\"])) # Our queries (qid => question)\n",
"relevant_docs = {} # Query ID to relevant documents (qid => set([relevant_cids])\n",
"for qid, corpus_ids in zip(relevant_docs_data[\"query-id\"], relevant_docs_data[\"corpus-id\"]):\n",
" qid = str(qid)\n",
" corpus_ids = str(corpus_ids)\n",
" if qid not in relevant_docs:\n",
" relevant_docs[qid] = set()\n",
" relevant_docs[qid].add(corpus_ids)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Finally we are ready to do the evaluation."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Given queries, a corpus and a mapping with relevant documents, the InformationRetrievalEvaluator computes different IR metrics.\n",
"ir_evaluator = InformationRetrievalEvaluator(\n",
" queries=queries,\n",
" corpus=corpus,\n",
" relevant_docs=relevant_docs,\n",
" name=\"BeIR-quora-dev\",\n",
")\n",
"\n",
"results = ir_evaluator(model)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"name": "python",
"version": "3.12.2"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
docs/source/tutorial/4_Evaluation/4.4.1.ipynb
+467
View File
@@ -0,0 +1,467 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Evaluate on BEIR"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[BEIR](https://github.com/beir-cellar/beir) (Benchmarking-IR) is a heterogeneous evaluation benchmark for information retrieval. \n",
"It is designed for evaluating the performance of NLP-based retrieval models and widely used by research of modern embedding models."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 0. Installation"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"First install the libraries we are using:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"% pip install beir FlagEmbedding"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. Evaluate using BEIR"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"BEIR contains 18 datasets which can be downloaded from the [link](https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/), while 4 of them are private datasets that need appropriate licences. If you want to access to those 4 datasets, take a look at their [wiki](https://github.com/beir-cellar/beir/wiki/Datasets-available) for more information. Information collected and codes adapted from BEIR GitHub [repo](https://github.com/beir-cellar/beir)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"| Dataset Name | Type | Queries | Documents | Avg. Docs/Q | Public | \n",
"| ---------| :-----------: | ---------| --------- | ------| :------------:| \n",
"| ``msmarco`` | `Train` `Dev` `Test` | 6,980 | 8.84M | 1.1 | Yes | \n",
"| ``trec-covid``| `Test` | 50| 171K| 493.5 | Yes | \n",
"| ``nfcorpus`` | `Train` `Dev` `Test` | 323 | 3.6K | 38.2 | Yes |\n",
"| ``bioasq``| `Train` `Test` | 500 | 14.91M | 8.05 | No | \n",
"| ``nq``| `Train` `Test` | 3,452 | 2.68M | 1.2 | Yes | \n",
"| ``hotpotqa``| `Train` `Dev` `Test` | 7,405 | 5.23M | 2.0 | Yes |\n",
"| ``fiqa`` | `Train` `Dev` `Test` | 648 | 57K | 2.6 | Yes | \n",
"| ``signal1m`` | `Test` | 97 | 2.86M | 19.6 | No |\n",
"| ``trec-news`` | `Test` | 57 | 595K | 19.6 | No |\n",
"| ``arguana`` | `Test` | 1,406 | 8.67K | 1.0 | Yes |\n",
"| ``webis-touche2020``| `Test` | 49 | 382K | 49.2 | Yes |\n",
"| ``cqadupstack``| `Test` | 13,145 | 457K | 1.4 | Yes |\n",
"| ``quora``| `Dev` `Test` | 10,000 | 523K | 1.6 | Yes | \n",
"| ``dbpedia-entity``| `Dev` `Test` | 400 | 4.63M | 38.2 | Yes | \n",
"| ``scidocs``| `Test` | 1,000 | 25K | 4.9 | Yes | \n",
"| ``fever``| `Train` `Dev` `Test` | 6,666 | 5.42M | 1.2| Yes | \n",
"| ``climate-fever``| `Test` | 1,535 | 5.42M | 3.0 | Yes |\n",
"| ``scifact``| `Train` `Test` | 300 | 5K | 1.1 | Yes |"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 1.1 Load Dataset"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"First prepare the logging setup."
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [],
"source": [
"import logging\n",
"from beir import LoggingHandler\n",
"\n",
"logging.basicConfig(format='%(message)s',\n",
" level=logging.INFO,\n",
" handlers=[LoggingHandler()])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In this demo, we choose the `arguana` dataset for a quick demonstration."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Dataset downloaded here: /share/project/xzy/Projects/FlagEmbedding/Tutorials/4_Evaluation/data/arguana\n"
]
}
],
"source": [
"import os\n",
"from beir import util\n",
"\n",
"url = \"https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/arguana.zip\"\n",
"out_dir = os.path.join(os.getcwd(), \"data\")\n",
"data_path = util.download_and_unzip(url, out_dir)\n",
"print(f\"Dataset is stored at: {data_path}\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2024-11-15 03:54:55,809 - Loading Corpus...\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"100%|██████████| 8674/8674 [00:00<00:00, 158928.31it/s]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"2024-11-15 03:54:55,891 - Loaded 8674 TEST Documents.\n",
"2024-11-15 03:54:55,891 - Doc Example: {'text': \"You don’t have to be vegetarian to be green. Many special environments have been created by livestock farming – for example chalk down land in England and mountain pastures in many countries. Ending livestock farming would see these areas go back to woodland with a loss of many unique plants and animals. Growing crops can also be very bad for the planet, with fertilisers and pesticides polluting rivers, lakes and seas. Most tropical forests are now cut down for timber, or to allow oil palm trees to be grown in plantations, not to create space for meat production. British farmer and former editor Simon Farrell also states: “Many vegans and vegetarians rely on one source from the U.N. calculation that livestock generates 18% of global carbon emissions, but this figure contains basic mistakes. It attributes all deforestation from ranching to cattle, rather than logging or development. It also muddles up one-off emissions from deforestation with on-going pollution.” He also refutes the statement of meat production inefficiency: “Scientists have calculated that globally the ratio between the amounts of useful plant food used to produce meat is about 5 to 1. If you feed animals only food that humans can eat — which is, indeed, largely the case in the Western world — that may be true. But animals also eat food we can't eat, such as grass. So the real conversion figure is 1.4 to 1.” [1] At the same time eating a vegetarian diet may be no more environmentally friendly than a meat based diet if it is not sustainably sourced or uses perishable fruit and vegetables that are flown in from around the world. Eating locally sourced food can has as big an impact as being vegetarian. [2] [1] Tara Kelly, Simon Fairlie: How Eating Meat Can Save the World, 12 October 2010 [2] Lucy Siegle, ‘It is time to become a vegetarian?’ The Observer, 18th May 2008\", 'title': 'animals environment general health health general weight philosophy ethics'}\n",
"2024-11-15 03:54:55,891 - Loading Queries...\n",
"2024-11-15 03:54:55,903 - Loaded 1406 TEST Queries.\n",
"2024-11-15 03:54:55,903 - Query Example: Being vegetarian helps the environment Becoming a vegetarian is an environmentally friendly thing to do. Modern farming is one of the main sources of pollution in our rivers. Beef farming is one of the main causes of deforestation, and as long as people continue to buy fast food in their billions, there will be a financial incentive to continue cutting down trees to make room for cattle. Because of our desire to eat fish, our rivers and seas are being emptied of fish and many species are facing extinction. Energy resources are used up much more greedily by meat farming than my farming cereals, pulses etc. Eating meat and fish not only causes cruelty to animals, it causes serious harm to the environment and to biodiversity. For example consider Meat production related pollution and deforestation At Toronto’s 1992 Royal Agricultural Winter Fair, Agriculture Canada displayed two contrasting statistics: “it takes four football fields of land (about 1.6 hectares) to feed each Canadian” and “one apple tree produces enough fruit to make 320 pies.” Think about it — a couple of apple trees and a few rows of wheat on a mere fraction of a hectare could produce enough food for one person! [1] The 2006 U.N. Food and Agriculture Organization (FAO) report concluded that worldwide livestock farming generates 18% of the planet's greenhouse gas emissions — by comparison, all the world's cars, trains, planes and boats account for a combined 13% of greenhouse gas emissions. [2] As a result of the above point producing meat damages the environment. The demand for meat drives deforestation. Daniel Cesar Avelino of Brazil's Federal Public Prosecution Office says “We know that the single biggest driver of deforestation in the Amazon is cattle.” This clearing of tropical rainforests such as the Amazon for agriculture is estimated to produce 17% of the world's greenhouse gas emissions. [3] Not only this but the production of meat takes a lot more energy than it ultimately gives us chicken meat production consumes energy in a 4:1 ratio to protein output; beef cattle production requires an energy input to protein output ratio of 54:1. The same is true with water use due to the same phenomenon of meat being inefficient to produce in terms of the amount of grain needed to produce the same weight of meat, production requires a lot of water. Water is another scarce resource that we will soon not have enough of in various areas of the globe. Grain-fed beef production takes 100,000 liters of water for every kilogram of food. Raising broiler chickens takes 3,500 liters of water to make a kilogram of meat. In comparison, soybean production uses 2,000 liters for kilogram of food produced; rice, 1,912; wheat, 900; and potatoes, 500 liters. [4] This is while there are areas of the globe that have severe water shortages. With farming using up to 70 times more water than is used for domestic purposes: cooking and washing. A third of the population of the world is already suffering from a shortage of water. [5] Groundwater levels are falling all over the world and rivers are beginning to dry up. Already some of the biggest rivers such as China’s Yellow river do not reach the sea. [6] With a rising population becoming vegetarian is the only responsible way to eat. [1] Stephen Leckie, ‘How Meat-centred Eating Patterns Affect Food Security and the Environment’, International development research center [2] Bryan Walsh, Meat: Making Global Warming Worse, Time magazine, 10 September 2008 . [3] David Adam, Supermarket suppliers ‘helping to destroy Amazon rainforest’, The Guardian, 21st June 2009. [4] Roger Segelken, U.S. could feed 800 million people with grain that livestock eat, Cornell Science News, 7th August 1997. [5] Fiona Harvey, Water scarcity affects one in three, FT.com, 21st August 2003 [6] Rupert Wingfield-Hayes, Yellow river ‘drying up’, BBC News, 29th July 2004\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\n"
]
}
],
"source": [
"from beir.datasets.data_loader import GenericDataLoader\n",
"\n",
"corpus, queries, qrels = GenericDataLoader(\"data/arguana\").load(split=\"test\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 1.2 Evaluation"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Then we load `bge-base-en-v1.5` from huggingface and evaluate its performance on arguana."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2024-11-15 04:00:45,253 - Use pytorch device_name: cuda\n",
"2024-11-15 04:00:45,254 - Load pretrained SentenceTransformer: BAAI/bge-base-en-v1.5\n",
"2024-11-15 04:00:48,750 - Encoding Queries...\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Batches: 100%|██████████| 11/11 [00:01<00:00, 8.27it/s]\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"2024-11-15 04:00:50,177 - Sorting Corpus by document length (Longest first)...\n",
"2024-11-15 04:00:50,183 - Encoding Corpus in batches... Warning: This might take a while!\n",
"2024-11-15 04:00:50,183 - Scoring Function: Cosine Similarity (cos_sim)\n",
"2024-11-15 04:00:50,184 - Encoding Batch 1/1...\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Batches: 100%|██████████| 68/68 [00:07<00:00, 9.43it/s]\n"
]
}
],
"source": [
"from beir.retrieval.evaluation import EvaluateRetrieval\n",
"from beir.retrieval import models\n",
"from beir.retrieval.search.dense import DenseRetrievalExactSearch as DRES\n",
"\n",
"\n",
"# Load bge model using Sentence Transformers\n",
"model = DRES(models.SentenceBERT(\"BAAI/bge-base-en-v1.5\"), batch_size=128)\n",
"retriever = EvaluateRetrieval(model, score_function=\"cos_sim\")\n",
"\n",
"# Get the searching results\n",
"results = retriever.retrieve(corpus, queries)"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2024-11-15 04:00:58,514 - Retriever evaluation for k in: [1, 3, 5, 10, 100, 1000]\n",
"2024-11-15 04:00:58,514 - For evaluation, we ignore identical query and document ids (default), please explicitly set ``ignore_identical_ids=False`` to ignore this.\n",
"2024-11-15 04:00:59,184 - \n",
"\n",
"2024-11-15 04:00:59,188 - NDCG@1: 0.4075\n",
"2024-11-15 04:00:59,188 - NDCG@3: 0.5572\n",
"2024-11-15 04:00:59,188 - NDCG@5: 0.5946\n",
"2024-11-15 04:00:59,188 - NDCG@10: 0.6361\n",
"2024-11-15 04:00:59,188 - NDCG@100: 0.6606\n",
"2024-11-15 04:00:59,188 - NDCG@1000: 0.6613\n",
"2024-11-15 04:00:59,188 - \n",
"\n",
"2024-11-15 04:00:59,188 - MAP@1: 0.4075\n",
"2024-11-15 04:00:59,188 - MAP@3: 0.5193\n",
"2024-11-15 04:00:59,188 - MAP@5: 0.5402\n",
"2024-11-15 04:00:59,188 - MAP@10: 0.5577\n",
"2024-11-15 04:00:59,188 - MAP@100: 0.5634\n",
"2024-11-15 04:00:59,188 - MAP@1000: 0.5635\n",
"2024-11-15 04:00:59,188 - \n",
"\n",
"2024-11-15 04:00:59,188 - Recall@1: 0.4075\n",
"2024-11-15 04:00:59,188 - Recall@3: 0.6671\n",
"2024-11-15 04:00:59,188 - Recall@5: 0.7575\n",
"2024-11-15 04:00:59,188 - Recall@10: 0.8841\n",
"2024-11-15 04:00:59,188 - Recall@100: 0.9915\n",
"2024-11-15 04:00:59,189 - Recall@1000: 0.9964\n",
"2024-11-15 04:00:59,189 - \n",
"\n",
"2024-11-15 04:00:59,189 - P@1: 0.4075\n",
"2024-11-15 04:00:59,189 - P@3: 0.2224\n",
"2024-11-15 04:00:59,189 - P@5: 0.1515\n",
"2024-11-15 04:00:59,189 - P@10: 0.0884\n",
"2024-11-15 04:00:59,189 - P@100: 0.0099\n",
"2024-11-15 04:00:59,189 - P@1000: 0.0010\n"
]
}
],
"source": [
"logging.info(\"Retriever evaluation for k in: {}\".format(retriever.k_values))\n",
"ndcg, _map, recall, precision = retriever.evaluate(qrels, results, retriever.k_values)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2. Evaluate using FlagEmbedding"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We provide independent evaluation for popular datasets and benchmarks. Try the following code to run the evaluation, or run the shell script provided in [example](../../examples/evaluation/beir/eval_beir.sh) folder."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Load the arguments:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"import sys\n",
"\n",
"arguments = \"\"\"-\n",
" --eval_name beir \n",
" --dataset_dir ./beir/data \n",
" --dataset_names arguana\n",
" --splits test dev \n",
" --corpus_embd_save_dir ./beir/corpus_embd \n",
" --output_dir ./beir/search_results \n",
" --search_top_k 1000 \n",
" --rerank_top_k 100 \n",
" --cache_path /root/.cache/huggingface/hub \n",
" --overwrite True \n",
" --k_values 10 100 \n",
" --eval_output_method markdown \n",
" --eval_output_path ./beir/beir_eval_results.md \n",
" --eval_metrics ndcg_at_10 recall_at_100 \n",
" --ignore_identical_ids True \n",
" --embedder_name_or_path BAAI/bge-base-en-v1.5 \n",
" --embedder_batch_size 1024\n",
" --devices cuda:4\n",
"\"\"\".replace('\\n','')\n",
"\n",
"sys.argv = arguments.split()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Then pass the arguments to HFArgumentParser and run the evaluation."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Split 'dev' not found in the dataset. Removing it from the list.\n",
"ignore_identical_ids is set to True. This means that the search results will not contain identical ids. Note: Dataset such as MIRACL should NOT set this to True.\n",
"pre tokenize: 100%|██████████| 9/9 [00:00<00:00, 16.19it/s]\n",
"You're using a BertTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.\n",
"Inference Embeddings: 100%|██████████| 9/9 [00:11<00:00, 1.27s/it]\n",
"pre tokenize: 100%|██████████| 2/2 [00:00<00:00, 19.54it/s]\n",
"Inference Embeddings: 100%|██████████| 2/2 [00:02<00:00, 1.29s/it]\n",
"Searching: 100%|██████████| 44/44 [00:00<00:00, 208.73it/s]\n"
]
}
],
"source": [
"from transformers import HfArgumentParser\n",
"\n",
"from FlagEmbedding.evaluation.beir import (\n",
" BEIREvalArgs, BEIREvalModelArgs,\n",
" BEIREvalRunner\n",
")\n",
"\n",
"\n",
"parser = HfArgumentParser((\n",
" BEIREvalArgs,\n",
" BEIREvalModelArgs\n",
"))\n",
"\n",
"eval_args, model_args = parser.parse_args_into_dataclasses()\n",
"eval_args: BEIREvalArgs\n",
"model_args: BEIREvalModelArgs\n",
"\n",
"runner = BEIREvalRunner(\n",
" eval_args=eval_args,\n",
" model_args=model_args\n",
")\n",
"\n",
"runner.run()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Take a look at the results and choose the way you prefer!"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{\n",
" \"arguana-test\": {\n",
" \"ndcg_at_10\": 0.63668,\n",
" \"ndcg_at_100\": 0.66075,\n",
" \"map_at_10\": 0.55801,\n",
" \"map_at_100\": 0.56358,\n",
" \"recall_at_10\": 0.88549,\n",
" \"recall_at_100\": 0.99147,\n",
" \"precision_at_10\": 0.08855,\n",
" \"precision_at_100\": 0.00991,\n",
" \"mrr_at_10\": 0.55809,\n",
" \"mrr_at_100\": 0.56366\n",
" }\n",
"}\n"
]
}
],
"source": [
"with open('beir/search_results/bge-base-en-v1.5/NoReranker/EVAL/eval_results.json', 'r') as content_file:\n",
" print(content_file.read())"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "dev",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.7"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
docs/source/tutorial/4_Evaluation/4.5.1.ipynb
+738
View File
@@ -0,0 +1,738 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Evaluate on MIRACL"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[MIRACL](https://project-miracl.github.io/) (Multilingual Information Retrieval Across a Continuum of Languages) is an WSDM 2023 Cup challenge that focuses on search across 18 different languages. They release a multilingual retrieval dataset containing the train and dev set for 16 “known languages” and only dev set for 2 “surprise languages”. The topics are generated by native speakers of each language, who also label the relevance between the topics and a given document list. You can found the dataset on HuggingFace."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Note: We highly recommend you to run the evaluation of MIRACL on GPU. For reference, it takes about an hour for the whole process on a 8xA100 40G node."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 0. Installation"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"First install the libraries we are using:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"% pip install FlagEmbedding pytrec_eval"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. Dataset"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"With the great number of passages and articles in the 18 languages. MIRACL is a resourceful dataset for training or evaluating multi-lingual model. The data can be downloaded from [Hugging Face](https://huggingface.co/datasets/miracl/miracl)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"| Language | # of Passages | # of Articles |\n",
"|:----------------|--------------:|--------------:|\n",
"| Arabic (ar) | 2,061,414 | 656,982 |\n",
"| Bengali (bn) | 297,265 | 63,762 |\n",
"| English (en) | 32,893,221 | 5,758,285 |\n",
"| Spanish (es) | 10,373,953 | 1,669,181 |\n",
"| Persian (fa) | 2,207,172 | 857,827 |\n",
"| Finnish (fi) | 1,883,509 | 447,815 |\n",
"| French (fr) | 14,636,953 | 2,325,608 |\n",
"| Hindi (hi) | 506,264 | 148,107 |\n",
"| Indonesian (id) | 1,446,315 | 446,330 |\n",
"| Japanese (ja) | 6,953,614 | 1,133,444 |\n",
"| Korean (ko) | 1,486,752 | 437,373 |\n",
"| Russian (ru) | 9,543,918 | 1,476,045 |\n",
"| Swahili (sw) | 131,924 | 47,793 |\n",
"| Telugu (te) | 518,079 | 66,353 |\n",
"| Thai (th) | 542,166 | 128,179 |\n",
"| Chinese (zh) | 4,934,368 | 1,246,389 |"
]
},
{
"cell_type": "code",
"execution_count": 38,
"metadata": {},
"outputs": [],
"source": [
"from datasets import load_dataset\n",
"\n",
"lang = \"en\"\n",
"corpus = load_dataset(\"miracl/miracl-corpus\", lang, trust_remote_code=True)['train']"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Each passage in the corpus has three parts: `docid`, `title`, and `text`. In the structure of document with docid `x#y`, `x` indicates the id of Wikipedia article, and `y` is the number of passage within that article. The title is the name of the article with id `x` that passage belongs to. The text is the text body of the passage."
]
},
{
"cell_type": "code",
"execution_count": 39,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'docid': '56672809#4',\n",
" 'title': 'Glen Tomasetti',\n",
" 'text': 'In 1967 Tomasetti was prosecuted after refusing to pay one sixth of her taxes on the grounds that one sixth of the federal budget was funding Australia\\'s military presence in Vietnam. In court she argued that Australia\\'s participation in the Vietnam War violated its international legal obligations as a member of the United Nations. Public figures such as Joan Baez had made similar protests in the USA, but Tomasetti\\'s prosecution was \"believed to be the first case of its kind in Australia\", according to a contemporary news report. Tomasetti was eventually ordered to pay the unpaid taxes.'}"
]
},
"execution_count": 39,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"corpus[0]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The qrels have following form:"
]
},
{
"cell_type": "code",
"execution_count": 40,
"metadata": {},
"outputs": [],
"source": [
"dev = load_dataset('miracl/miracl', lang, trust_remote_code=True)['dev']"
]
},
{
"cell_type": "code",
"execution_count": 41,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'query_id': '0',\n",
" 'query': 'Is Creole a pidgin of French?',\n",
" 'positive_passages': [{'docid': '462221#4',\n",
" 'text': \"At the end of World War II in 1945, Korea was divided into North Korea and South Korea with North Korea (assisted by the Soviet Union), becoming a communist government after 1946, known as the Democratic People's Republic, followed by South Korea becoming the Republic of Korea. China became the communist People's Republic of China in 1949. In 1950, the Soviet Union backed North Korea while the United States backed South Korea, and China allied with the Soviet Union in what was to become the first military action of the Cold War.\",\n",
" 'title': 'Eighth United States Army'},\n",
" {'docid': '29810#23',\n",
" 'text': 'The large size of Texas and its location at the intersection of multiple climate zones gives the state highly variable weather. The Panhandle of the state has colder winters than North Texas, while the Gulf Coast has mild winters. Texas has wide variations in precipitation patterns. El Paso, on the western end of the state, averages of annual rainfall, while parts of southeast Texas average as much as per year. Dallas in the North Central region averages a more moderate per year.',\n",
" 'title': 'Texas'},\n",
" {'docid': '3716905#0',\n",
" 'text': 'A French creole, or French-based creole language, is a creole language (contact language with native speakers) for which French is the \"lexifier\". Most often this lexifier is not modern French but rather a 17th-century koiné of French from Paris, the French Atlantic harbors, and the nascent French colonies. French-based creole languages are spoken natively by millions of people worldwide, primarily in the Americas and on archipelagos throughout the Indian Ocean. This article also contains information on French pidgin languages, contact languages that lack native speakers.',\n",
" 'title': 'French-based creole languages'},\n",
" {'docid': '22399755#18',\n",
" 'text': 'There are many hypotheses on the origins of Haitian Creole. Linguist John Singler suggests that it most likely emerged under French control in colonial years when shifted its economy focused heavily on sugar production. This resulted in a much larger population of enslaved Africans, whose interaction with the French created the circumstances for the dialect to evolve from a pidgin to a Creole. His research and the research of Claire Lefebvre of the Université du Québec à Montréal suggests that Creole, despite drawing 90% of its lexicon from French, is the syntactic cousin of Fon, a Gbe language of the Niger-Congo family spoken in Benin. At the time of the emergence of Haitian Creole, 50% of the enslaved Africans in Haiti were Gbe speakers.',\n",
" 'title': 'Haitian literature'}],\n",
" 'negative_passages': [{'docid': '1170520#2',\n",
" 'text': 'Louisiana Creole is a contact language that arose in the 18th century from interactions between speakers of the lexifier language of Standard French and several substrate or adstrate languages from Africa. Prior to its establishment as a Creole, the precursor was considered a pidgin language. The social situation that gave rise to the Louisiana Creole language was unique, in that the lexifier language was the language found at the contact site. More often the lexifier is the language that arrives at the contact site belonging to the substrate/adstrate languages. Neither the French, the French-Canadians, nor the African slaves were native to the area; this fact categorizes Louisiana Creole as a contact language that arose between exogenous ethnicities. Once the pidgin tongue was transmitted to the next generation as a \"lingua franca\" (who were then considered the first native speakers of the new grammar), it could effectively be classified as a creole language.',\n",
" 'title': 'Louisiana Creole'},\n",
" {'docid': '49823#1',\n",
" 'text': 'The precise number of creole languages is not known, particularly as many are poorly attested or documented. About one hundred creole languages have arisen since 1500. These are predominantly based on European languages such as English and French due to the European Age of Discovery and the Atlantic slave trade that arose at that time. With the improvements in ship-building and navigation, traders had to learn to communicate with people around the world, and the quickest way to do this was to develop a pidgin, or simplified language suited to the purpose; in turn, full creole languages developed from these pidgins. In addition to creoles that have European languages as their base, there are, for example, creoles based on Arabic, Chinese, and Malay. The creole with the largest number of speakers is Haitian Creole, with almost ten million native speakers, followed by Tok Pisin with about 4 million, most of whom are second-language speakers.',\n",
" 'title': 'Creole language'},\n",
" {'docid': '1651722#10',\n",
" 'text': 'Krio is an English-based creole from which descend Nigerian Pidgin English and Cameroonian Pidgin English and Pichinglis. It is also similar to English-based creole languages spoken in the Americas, especially the Gullah language, Jamaican Patois (Jamaican Creole), and Bajan Creole but it has its own distinctive character. It also shares some linguistic similarities with non-English creoles, such as the French-based creole languages in the Caribbean.',\n",
" 'title': 'Krio language'},\n",
" {'docid': '540382#4',\n",
" 'text': 'Until recently creoles were considered \"degenerate\" dialects of Portuguese unworthy of attention. As a consequence, there is little documentation on the details of their formation. Since the 20th century, increased study of creoles by linguists led to several theories being advanced. The monogenetic theory of pidgins assumes that some type of pidgin language — dubbed West African Pidgin Portuguese — based on Portuguese was spoken from the 15th to 18th centuries in the forts established by the Portuguese on the West African coast. According to this theory, this variety may have been the starting point of all the pidgin and creole languages. This may explain to some extent why Portuguese lexical items can be found in many creoles, but more importantly, it would account for the numerous grammatical similarities shared by such languages, such as the preposition \"na\", meaning \"in\" and/or \"on\", which would come from the Portuguese contraction \"na\" meaning \"in the\" (feminine singular).',\n",
" 'title': 'Portuguese-based creole languages'},\n",
" {'docid': '49823#7',\n",
" 'text': 'Other scholars, such as Salikoko Mufwene, argue that pidgins and creoles arise independently under different circumstances, and that a pidgin need not always precede a creole nor a creole evolve from a pidgin. Pidgins, according to Mufwene, emerged in trade colonies among \"users who preserved their native vernaculars for their day-to-day interactions.\" Creoles, meanwhile, developed in settlement colonies in which speakers of a European language, often indentured servants whose language would be far from the standard in the first place, interacted extensively with non-European slaves, absorbing certain words and features from the slaves\\' non-European native languages, resulting in a heavily basilectalized version of the original language. These servants and slaves would come to use the creole as an everyday vernacular, rather than merely in situations in which contact with a speaker of the superstrate was necessary.',\n",
" 'title': 'Creole language'},\n",
" {'docid': '11236157#2',\n",
" 'text': 'While many creoles around the world have lexicons based on languages other than Portuguese (e.g. English, French, Spanish, Dutch), it was hypothesized that such creoles were derived from this lingua franca by means of relexification, i.e. the process in which a pidgin or creole incorporates a significant amount of its lexicon from another language while keeping the grammar intact. There is some evidence that relexification is a real process. Pieter Muysken and show that there are languages which derive their grammar and lexicon from two different languages respectively, which could be easily explained with the relexification hypothesis. Also, Saramaccan seems to be a pidgin frozen in the middle of relexification from Portuguese to English. However, in cases of such mixed languages, as call them, there is never a one-to-one relationship between the grammar or lexicon of the mixed language and the grammar or lexicon of the language they attribute it to.',\n",
" 'title': 'Monogenetic theory of pidgins'},\n",
" {'docid': '1612877#8',\n",
" 'text': 'A mixed language differs from pidgins, creoles and code-switching in very fundamental ways. In most cases, mixed language speakers are fluent, even native, speakers of both languages; however, speakers of Michif (a N-V mixed language) are unique in that many are not fluent in both of the sources languages. Pidgins, on the other hand, develop in a situation, usually in the context of trade, where speakers of two (or more) different languages come into contact and need to find some way to communicate with each other. Creoles develop when a pidgin language becomes a first language for young speakers. While creoles tend to have drastically simplified morphologies, mixed languages often retain the inflectional complexities of one, or both, of parent languages. For instance, Michif retains the complexities of its French nouns and its Cree verbs.',\n",
" 'title': 'Mixed language'},\n",
" {'docid': '9606120#4',\n",
" 'text': 'While it is classified as a pidgin language, this is inaccurate. Speakers are already fluent in either English and French, and as such it is not used in situations where both parties lack a common tongue. As a whole, Camfranglais sets itself apart from other pidgins and creoles in that it consists of an array of languages, at least one of which is already known by those speaking it. For instance, while it contains elements of borrowing, code-switching, and pidgin languages, it is not a contact language as both parties can be presumed to speak French, the lexifer. Numerous other classifications have been proposed, like ‘pidgin’, ‘argot’, ‘youth language’, a ‘sabir camerounais’, an ‘appropriation vernaculaire du français’ or a ‘hybrid slang’. However, as Camfranglais is more developed than a slang, this too is insufficient. Kießling proposes it be classified as a \\'highly hybrid sociolect of the urban youth type\", a definition that Stein-Kanjora agrees with.',\n",
" 'title': 'Camfranglais'}]}"
]
},
"execution_count": 41,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"dev[0]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Each item has four parts: `query_id`, `query`, `positive_passages`, and `negative_passages`. Here, `query_id` and `query` correspond to the id and text content of the qeury. `positive_passages` and `negative_passages` are list of passages with their corresponding `docid`, `title`, and `text`. \n",
"\n",
"This structure is the same in the `train`, `dev`, `testA` and `testB` sets."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Then we process the ids and text of queries and corpus, and get the qrels of the dev set."
]
},
{
"cell_type": "code",
"execution_count": 42,
"metadata": {},
"outputs": [],
"source": [
"corpus_ids = corpus['docid']\n",
"corpus_text = []\n",
"for doc in corpus:\n",
" corpus_text.append(f\"{doc['title']} {doc['text']}\".strip())\n",
"\n",
"queries_ids = dev['query_id']\n",
"queries_text = dev['query']"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2. Evaluate from scratch"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 2.1 Embedding"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In the demo we use bge-base-en-v1.5, feel free to change to the model you prefer."
]
},
{
"cell_type": "code",
"execution_count": 43,
"metadata": {},
"outputs": [],
"source": [
"import os \n",
"os.environ['TRANSFORMERS_NO_ADVISORY_WARNINGS'] = 'true'\n",
"os.environ['SETUPTOOLS_USE_DISTUTILS'] = ''"
]
},
{
"cell_type": "code",
"execution_count": 44,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"initial target device: 100%|██████████| 8/8 [00:29<00:00, 3.66s/it]\n",
"pre tokenize: 100%|██████████| 1/1 [00:00<00:00, 52.84it/s]\n",
"pre tokenize: 100%|██████████| 1/1 [00:00<00:00, 55.15it/s]\n",
"pre tokenize: 100%|██████████| 1/1 [00:00<00:00, 56.49it/s]\n",
"pre tokenize: 100%|██████████| 1/1 [00:00<00:00, 55.22it/s]\n",
"pre tokenize: 100%|██████████| 1/1 [00:00<00:00, 49.22it/s]\n",
"pre tokenize: 100%|██████████| 1/1 [00:00<00:00, 54.69it/s]\n",
"pre tokenize: 100%|██████████| 1/1 [00:00<00:00, 49.16it/s]\n",
"pre tokenize: 100%|██████████| 1/1 [00:00<00:00, 50.77it/s]\n",
"Chunks: 100%|██████████| 8/8 [00:10<00:00, 1.27s/it]\n",
"pre tokenize: 100%|██████████| 16062/16062 [08:12<00:00, 32.58it/s] \n",
"pre tokenize: 100%|██████████| 16062/16062 [08:44<00:00, 30.60it/s]68s/it]\n",
"pre tokenize: 100%|██████████| 16062/16062 [08:39<00:00, 30.90it/s]41s/it]\n",
"pre tokenize: 100%|██████████| 16062/16062 [09:04<00:00, 29.49it/s]43s/it]\n",
"pre tokenize: 100%|██████████| 16062/16062 [09:27<00:00, 28.29it/s]it/s]t]\n",
"pre tokenize: 100%|██████████| 16062/16062 [09:08<00:00, 29.30it/s]32s/it]\n",
"pre tokenize: 100%|██████████| 16062/16062 [08:59<00:00, 29.77it/s]it/s]t]\n",
"pre tokenize: 100%|██████████| 16062/16062 [09:04<00:00, 29.50it/s]29s/it]\n",
"Inference Embeddings: 100%|██████████| 16062/16062 [17:10<00:00, 15.59it/s] \n",
"Inference Embeddings: 100%|██████████| 16062/16062 [17:04<00:00, 15.68it/s]]\n",
"Inference Embeddings: 100%|██████████| 16062/16062 [17:01<00:00, 15.72it/s]s]\n",
"Inference Embeddings: 100%|██████████| 16062/16062 [17:28<00:00, 15.32it/s]\n",
"Inference Embeddings: 100%|██████████| 16062/16062 [17:43<00:00, 15.10it/s]\n",
"Inference Embeddings: 100%|██████████| 16062/16062 [17:27<00:00, 15.34it/s]\n",
"Inference Embeddings: 100%|██████████| 16062/16062 [17:36<00:00, 15.20it/s]\n",
"Inference Embeddings: 100%|██████████| 16062/16062 [17:31<00:00, 15.28it/s]\n",
"Chunks: 100%|██████████| 8/8 [27:49<00:00, 208.64s/it]\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"shape of the embeddings: (32893221, 768)\n",
"data type of the embeddings: float16\n"
]
}
],
"source": [
"from FlagEmbedding import FlagModel\n",
"\n",
"# get the BGE embedding model\n",
"model = FlagModel('BAAI/bge-base-en-v1.5')\n",
"\n",
"# get the embedding of the queries and corpus\n",
"queries_embeddings = model.encode_queries(queries_text)\n",
"corpus_embeddings = model.encode_corpus(corpus_text)\n",
"\n",
"print(\"shape of the embeddings:\", corpus_embeddings.shape)\n",
"print(\"data type of the embeddings: \", corpus_embeddings.dtype)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 2.2 Indexing"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Create a Faiss index to store the embeddings."
]
},
{
"cell_type": "code",
"execution_count": 45,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"total number of vectors: 32893221\n"
]
}
],
"source": [
"import faiss\n",
"import numpy as np\n",
"\n",
"# get the length of our embedding vectors, vectors by bge-base-en-v1.5 have length 768\n",
"dim = corpus_embeddings.shape[-1]\n",
"\n",
"# create the faiss index and store the corpus embeddings into the vector space\n",
"index = faiss.index_factory(dim, 'Flat', faiss.METRIC_INNER_PRODUCT)\n",
"corpus_embeddings = corpus_embeddings.astype(np.float32)\n",
"# train and add the embeddings to the index\n",
"index.train(corpus_embeddings)\n",
"index.add(corpus_embeddings)\n",
"\n",
"print(f\"total number of vectors: {index.ntotal}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 2.3 Searching"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Use the Faiss index to search for each query."
]
},
{
"cell_type": "code",
"execution_count": 46,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Searching: 100%|██████████| 25/25 [15:03<00:00, 36.15s/it]\n"
]
}
],
"source": [
"from tqdm import tqdm\n",
"\n",
"query_size = len(queries_embeddings)\n",
"\n",
"all_scores = []\n",
"all_indices = []\n",
"\n",
"for i in tqdm(range(0, query_size, 32), desc=\"Searching\"):\n",
" j = min(i + 32, query_size)\n",
" query_embedding = queries_embeddings[i: j]\n",
" score, indice = index.search(query_embedding.astype(np.float32), k=100)\n",
" all_scores.append(score)\n",
" all_indices.append(indice)\n",
"\n",
"all_scores = np.concatenate(all_scores, axis=0)\n",
"all_indices = np.concatenate(all_indices, axis=0)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Then map the search results back to the indices in the dataset."
]
},
{
"cell_type": "code",
"execution_count": 47,
"metadata": {},
"outputs": [],
"source": [
"results = {}\n",
"for idx, (scores, indices) in enumerate(zip(all_scores, all_indices)):\n",
" results[queries_ids[idx]] = {}\n",
" for score, index in zip(scores, indices):\n",
" if index != -1:\n",
" results[queries_ids[idx]][corpus_ids[index]] = float(score)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 2.4 Evaluating"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Download the qrels file for evaluation:"
]
},
{
"cell_type": "code",
"execution_count": 48,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"--2024-11-21 10:26:16-- https://hf-mirror.com/datasets/miracl/miracl/resolve/main/miracl-v1.0-en/qrels/qrels.miracl-v1.0-en-dev.tsv\n",
"Resolving hf-mirror.com (hf-mirror.com)... 153.121.57.40, 133.242.169.68, 160.16.199.204\n",
"Connecting to hf-mirror.com (hf-mirror.com)|153.121.57.40|:443... connected.\n",
"HTTP request sent, awaiting response... 200 OK\n",
"Length: 167817 (164K) [text/plain]\n",
"Saving to: ‘qrels.miracl-v1.0-en-dev.tsv’\n",
"\n",
" 0K .......... .......... .......... .......... .......... 30% 109K 1s\n",
" 50K .......... .......... .......... .......... .......... 61% 44.5K 1s\n",
" 100K .......... .......... .......... .......... .......... 91% 69.6K 0s\n",
" 150K .......... ... 100% 28.0K=2.8s\n",
"\n",
"2024-11-21 10:26:20 (58.6 KB/s) - ‘qrels.miracl-v1.0-en-dev.tsv’ saved [167817/167817]\n",
"\n"
]
},
{
"data": {
"text/plain": [
"0"
]
},
"execution_count": 48,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"endpoint = os.getenv('HF_ENDPOINT', 'https://huggingface.co')\n",
"file_name = \"qrels.miracl-v1.0-en-dev.tsv\"\n",
"qrel_url = f\"wget {endpoint}/datasets/miracl/miracl/resolve/main/miracl-v1.0-en/qrels/{file_name}\"\n",
"\n",
"os.system(qrel_url)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Read the qrels from the file:"
]
},
{
"cell_type": "code",
"execution_count": 49,
"metadata": {},
"outputs": [],
"source": [
"qrels_dict = {}\n",
"with open(file_name, \"r\", encoding=\"utf-8\") as f:\n",
" for line in f.readlines():\n",
" qid, _, docid, rel = line.strip().split(\"\\t\")\n",
" qid, docid, rel = str(qid), str(docid), int(rel)\n",
" if qid not in qrels_dict:\n",
" qrels_dict[qid] = {}\n",
" qrels_dict[qid][docid] = rel"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Finally, use [pytrec_eval](https://github.com/cvangysel/pytrec_eval) library to help us calculate the scores of selected metrics:"
]
},
{
"cell_type": "code",
"execution_count": 50,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"defaultdict(<class 'list'>, {'NDCG@10': 0.46073, 'NDCG@100': 0.54336})\n",
"defaultdict(<class 'list'>, {'Recall@10': 0.55972, 'Recall@100': 0.83827})\n"
]
}
],
"source": [
"import pytrec_eval\n",
"from collections import defaultdict\n",
"\n",
"ndcg_string = \"ndcg_cut.\" + \",\".join([str(k) for k in [10,100]])\n",
"recall_string = \"recall.\" + \",\".join([str(k) for k in [10,100]])\n",
"\n",
"evaluator = pytrec_eval.RelevanceEvaluator(\n",
" qrels_dict, {ndcg_string, recall_string}\n",
")\n",
"scores = evaluator.evaluate(results)\n",
"\n",
"all_ndcgs, all_recalls = defaultdict(list), defaultdict(list)\n",
"for query_id in scores.keys():\n",
" for k in [10,100]:\n",
" all_ndcgs[f\"NDCG@{k}\"].append(scores[query_id][\"ndcg_cut_\" + str(k)])\n",
" all_recalls[f\"Recall@{k}\"].append(scores[query_id][\"recall_\" + str(k)])\n",
"\n",
"ndcg, recall = (\n",
" all_ndcgs.copy(),\n",
" all_recalls.copy(),\n",
")\n",
"\n",
"for k in [10,100]:\n",
" ndcg[f\"NDCG@{k}\"] = round(sum(ndcg[f\"NDCG@{k}\"]) / len(scores), 5)\n",
" recall[f\"Recall@{k}\"] = round(sum(recall[f\"Recall@{k}\"]) / len(scores), 5)\n",
"\n",
"print(ndcg)\n",
"print(recall)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3. Evaluate using FlagEmbedding"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We provide independent evaluation for popular datasets and benchmarks. Try the following code to run the evaluation, or run the shell script provided in [example](../../examples/evaluation/miracl/eval_miracl.sh) folder."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import sys\n",
"\n",
"arguments = \"\"\"- \\\n",
" --eval_name miracl \\\n",
" --dataset_dir ./miracl/data \\\n",
" --dataset_names en \\\n",
" --splits dev \\\n",
" --corpus_embd_save_dir ./miracl/corpus_embd \\\n",
" --output_dir ./miracl/search_results \\\n",
" --search_top_k 100 \\\n",
" --cache_path ./cache/data \\\n",
" --overwrite True \\\n",
" --k_values 10 100 \\\n",
" --eval_output_method markdown \\\n",
" --eval_output_path ./miracl/miracl_eval_results.md \\\n",
" --eval_metrics ndcg_at_10 recall_at_100 \\\n",
" --embedder_name_or_path BAAI/bge-base-en-v1.5 \\\n",
" --devices cuda:0 cuda:1 \\\n",
" --embedder_batch_size 1024\n",
"\"\"\".replace('\\n','')\n",
"\n",
"sys.argv = arguments.split()"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/root/anaconda3/envs/dev/lib/python3.12/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
" from .autonotebook import tqdm as notebook_tqdm\n",
"initial target device: 100%|██████████| 2/2 [00:09<00:00, 4.98s/it]\n",
"pre tokenize: 100%|██████████| 16062/16062 [18:01<00:00, 14.85it/s] \n",
"You're using a BertTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.\n",
"/root/anaconda3/envs/dev/lib/python3.12/site-packages/_distutils_hack/__init__.py:54: UserWarning: Reliance on distutils from stdlib is deprecated. Users must rely on setuptools to provide the distutils module. Avoid importing distutils or import setuptools first, and avoid setting SETUPTOOLS_USE_DISTUTILS=stdlib. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml\n",
" warnings.warn(\n",
"pre tokenize: 100%|██████████| 16062/16062 [18:44<00:00, 14.29it/s]92s/it]\n",
"Inference Embeddings: 0%| | 42/16062 [00:54<8:28:19, 1.90s/it]You're using a BertTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.\n",
"Inference Embeddings: 0%| | 43/16062 [00:56<8:22:03, 1.88s/it]/root/anaconda3/envs/dev/lib/python3.12/site-packages/_distutils_hack/__init__.py:54: UserWarning: Reliance on distutils from stdlib is deprecated. Users must rely on setuptools to provide the distutils module. Avoid importing distutils or import setuptools first, and avoid setting SETUPTOOLS_USE_DISTUTILS=stdlib. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml\n",
" warnings.warn(\n",
"Inference Embeddings: 100%|██████████| 16062/16062 [48:29<00:00, 5.52it/s] \n",
"Inference Embeddings: 100%|██████████| 16062/16062 [48:55<00:00, 5.47it/s]\n",
"Chunks: 100%|██████████| 2/2 [1:10:57<00:00, 2128.54s/it] \n",
"pre tokenize: 100%|██████████| 1/1 [00:11<00:00, 11.06s/it]\n",
"pre tokenize: 100%|██████████| 1/1 [00:12<00:00, 12.72s/it]\n",
"Inference Embeddings: 100%|██████████| 1/1 [00:00<00:00, 32.15it/s]\n",
"Inference Embeddings: 100%|██████████| 1/1 [00:00<00:00, 39.80it/s]\n",
"Chunks: 100%|██████████| 2/2 [00:31<00:00, 15.79s/it]\n",
"Searching: 100%|██████████| 25/25 [00:00<00:00, 26.24it/s]\n",
"Qrels not found in ./miracl/data/en/dev_qrels.jsonl. Trying to download the qrels from the remote and save it to ./miracl/data/en.\n",
"--2024-11-20 13:00:40-- https://hf-mirror.com/datasets/miracl/miracl/resolve/main/miracl-v1.0-en/qrels/qrels.miracl-v1.0-en-dev.tsv\n",
"Resolving hf-mirror.com (hf-mirror.com)... 133.242.169.68, 153.121.57.40, 160.16.199.204\n",
"Connecting to hf-mirror.com (hf-mirror.com)|133.242.169.68|:443... connected.\n",
"HTTP request sent, awaiting response... 200 OK\n",
"Length: 167817 (164K) [text/plain]\n",
"Saving to: ‘./cache/data/miracl/qrels.miracl-v1.0-en-dev.tsv’\n",
"\n",
" 0K .......... .......... .......... .......... .......... 30% 336K 0s\n",
" 50K .......... .......... .......... .......... .......... 61% 678K 0s\n",
" 100K .......... .......... .......... .......... .......... 91% 362K 0s\n",
" 150K .......... ... 100% 39.8K=0.7s\n",
"\n",
"2024-11-20 13:00:42 (231 KB/s) - ‘./cache/data/miracl/qrels.miracl-v1.0-en-dev.tsv’ saved [167817/167817]\n",
"\n",
"Loading and Saving qrels: 100%|██████████| 8350/8350 [00:00<00:00, 184554.95it/s]\n"
]
}
],
"source": [
"from transformers import HfArgumentParser\n",
"\n",
"from FlagEmbedding.evaluation.miracl import (\n",
" MIRACLEvalArgs, MIRACLEvalModelArgs,\n",
" MIRACLEvalRunner\n",
")\n",
"\n",
"\n",
"parser = HfArgumentParser((\n",
" MIRACLEvalArgs,\n",
" MIRACLEvalModelArgs\n",
"))\n",
"\n",
"eval_args, model_args = parser.parse_args_into_dataclasses()\n",
"eval_args: MIRACLEvalArgs\n",
"model_args: MIRACLEvalModelArgs\n",
"\n",
"runner = MIRACLEvalRunner(\n",
" eval_args=eval_args,\n",
" model_args=model_args\n",
")\n",
"\n",
"runner.run()"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{\n",
" \"en-dev\": {\n",
" \"ndcg_at_10\": 0.46053,\n",
" \"ndcg_at_100\": 0.54313,\n",
" \"map_at_10\": 0.35928,\n",
" \"map_at_100\": 0.38726,\n",
" \"recall_at_10\": 0.55972,\n",
" \"recall_at_100\": 0.83809,\n",
" \"precision_at_10\": 0.14018,\n",
" \"precision_at_100\": 0.02347,\n",
" \"mrr_at_10\": 0.54328,\n",
" \"mrr_at_100\": 0.54929\n",
" }\n",
"}\n"
]
}
],
"source": [
"with open('miracl/search_results/bge-base-en-v1.5/NoReranker/EVAL/eval_results.json', 'r') as content_file:\n",
" print(content_file.read())"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "dev",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.7"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
docs/source/tutorial/4_Evaluation/4.5.2.ipynb
+606
View File
File diff suppressed because one or more lines are too long