wehub-resource-sync
236 lines
8.3 KiB
Markdown
236 lines
8.3 KiB
Markdown
# Evaluation Result Reproduce
|
|
|
|
## Dataset
|
|
|
|
The dataset used in LightRAG can be downloaded from [TommyChien/UltraDomain](https://huggingface.co/datasets/TommyChien/UltraDomain).
|
|
|
|
## Generate Query
|
|
|
|
LightRAG uses the following prompt to generate high-level queries, with the corresponding code in `examples/generate_query.py`.
|
|
|
|
**Prompt**
|
|
|
|
```
|
|
Given the following description of a dataset:
|
|
|
|
{description}
|
|
|
|
Please identify 5 potential users who would engage with this dataset. For each user, list 5 tasks they would perform with this dataset. Then, for each (user, task) combination, generate 5 questions that require a high-level understanding of the entire dataset.
|
|
|
|
Output the results in the following structure:
|
|
- User 1: [user description]
|
|
- Task 1: [task description]
|
|
- Question 1:
|
|
- Question 2:
|
|
- Question 3:
|
|
- Question 4:
|
|
- Question 5:
|
|
- Task 2: [task description]
|
|
...
|
|
- Task 5: [task description]
|
|
- User 2: [user description]
|
|
...
|
|
- User 5: [user description]
|
|
...
|
|
```
|
|
|
|
## Batch Eval
|
|
|
|
To evaluate the performance of two RAG systems on high-level queries, LightRAG uses the following prompt, with the specific code available in `reproduce/batch_eval.py`.
|
|
|
|
**Prompt**
|
|
|
|
```
|
|
---Role---
|
|
You are an expert tasked with evaluating two answers to the same question based on three criteria: **Comprehensiveness**, **Diversity**, and **Empowerment**.
|
|
---Goal---
|
|
You will evaluate two answers to the same question based on three criteria: **Comprehensiveness**, **Diversity**, and **Empowerment**.
|
|
|
|
- **Comprehensiveness**: How much detail does the answer provide to cover all aspects and details of the question?
|
|
- **Diversity**: How varied and rich is the answer in providing different perspectives and insights on the question?
|
|
- **Empowerment**: How well does the answer help the reader understand and make informed judgments about the topic?
|
|
|
|
For each criterion, choose the better answer (either Answer 1 or Answer 2) and explain why. Then, select an overall winner based on these three categories.
|
|
|
|
Here is the question:
|
|
{query}
|
|
|
|
Here are the two answers:
|
|
|
|
**Answer 1:**
|
|
{answer1}
|
|
|
|
**Answer 2:**
|
|
{answer2}
|
|
|
|
Evaluate both answers using the three criteria listed above and provide detailed explanations for each criterion.
|
|
|
|
Output your evaluation in the following JSON format:
|
|
|
|
{{
|
|
"Comprehensiveness": {{
|
|
"Winner": "[Answer 1 or Answer 2]",
|
|
"Explanation": "[Provide explanation here]"
|
|
}},
|
|
"Empowerment": {{
|
|
"Winner": "[Answer 1 or Answer 2]",
|
|
"Explanation": "[Provide explanation here]"
|
|
}},
|
|
"Overall Winner": {{
|
|
"Winner": "[Answer 1 or Answer 2]",
|
|
"Explanation": "[Summarize why this answer is the overall winner based on the three criteria]"
|
|
}}
|
|
}}
|
|
```
|
|
|
|
## Overall Performance Table
|
|
|
|
||**Agriculture**||**CS**||**Legal**||**Mix**||
|
|
|----------------------|---------------|------------|------|------------|---------|------------|-------|------------|
|
|
||NaiveRAG|**LightRAG**|NaiveRAG|**LightRAG**|NaiveRAG|**LightRAG**|NaiveRAG|**LightRAG**|
|
|
|**Comprehensiveness**|32.4%|**67.6%**|38.4%|**61.6%**|16.4%|**83.6%**|38.8%|**61.2%**|
|
|
|**Diversity**|23.6%|**76.4%**|38.0%|**62.0%**|13.6%|**86.4%**|32.4%|**67.6%**|
|
|
|**Empowerment**|32.4%|**67.6%**|38.8%|**61.2%**|16.4%|**83.6%**|42.8%|**57.2%**|
|
|
|**Overall**|32.4%|**67.6%**|38.8%|**61.2%**|15.2%|**84.8%**|40.0%|**60.0%**|
|
|
||RQ-RAG|**LightRAG**|RQ-RAG|**LightRAG**|RQ-RAG|**LightRAG**|RQ-RAG|**LightRAG**|
|
|
|**Comprehensiveness**|31.6%|**68.4%**|38.8%|**61.2%**|15.2%|**84.8%**|39.2%|**60.8%**|
|
|
|**Diversity**|29.2%|**70.8%**|39.2%|**60.8%**|11.6%|**88.4%**|30.8%|**69.2%**|
|
|
|**Empowerment**|31.6%|**68.4%**|36.4%|**63.6%**|15.2%|**84.8%**|42.4%|**57.6%**|
|
|
|**Overall**|32.4%|**67.6%**|38.0%|**62.0%**|14.4%|**85.6%**|40.0%|**60.0%**|
|
|
||HyDE|**LightRAG**|HyDE|**LightRAG**|HyDE|**LightRAG**|HyDE|**LightRAG**|
|
|
|**Comprehensiveness**|26.0%|**74.0%**|41.6%|**58.4%**|26.8%|**73.2%**|40.4%|**59.6%**|
|
|
|**Diversity**|24.0%|**76.0%**|38.8%|**61.2%**|20.0%|**80.0%**|32.4%|**67.6%**|
|
|
|**Empowerment**|25.2%|**74.8%**|40.8%|**59.2%**|26.0%|**74.0%**|46.0%|**54.0%**|
|
|
|**Overall**|24.8%|**75.2%**|41.6%|**58.4%**|26.4%|**73.6%**|42.4%|**57.6%**|
|
|
||GraphRAG|**LightRAG**|GraphRAG|**LightRAG**|GraphRAG|**LightRAG**|GraphRAG|**LightRAG**|
|
|
|**Comprehensiveness**|45.6%|**54.4%**|48.4%|**51.6%**|48.4%|**51.6%**|**50.4%**|49.6%|
|
|
|**Diversity**|22.8%|**77.2%**|40.8%|**59.2%**|26.4%|**73.6%**|36.0%|**64.0%**|
|
|
|**Empowerment**|41.2%|**58.8%**|45.2%|**54.8%**|43.6%|**56.4%**|**50.8%**|49.2%|
|
|
|**Overall**|45.2%|**54.8%**|48.0%|**52.0%**|47.2%|**52.8%**|**50.4%**|49.6%|
|
|
|
|
## Reproduce
|
|
|
|
All the code can be found in the `./reproduce` directory.
|
|
|
|
### Step-0 Extract Unique Contexts
|
|
|
|
First, extract unique contexts from the datasets.
|
|
|
|
**Code**
|
|
|
|
```python
|
|
def extract_unique_contexts(input_directory, output_directory):
|
|
|
|
os.makedirs(output_directory, exist_ok=True)
|
|
|
|
jsonl_files = glob.glob(os.path.join(input_directory, '*.jsonl'))
|
|
print(f"Found {len(jsonl_files)} JSONL files.")
|
|
|
|
for file_path in jsonl_files:
|
|
filename = os.path.basename(file_path)
|
|
name, ext = os.path.splitext(filename)
|
|
output_filename = f"{name}_unique_contexts.json"
|
|
output_path = os.path.join(output_directory, output_filename)
|
|
|
|
unique_contexts_dict = {}
|
|
|
|
print(f"Processing file: {filename}")
|
|
|
|
try:
|
|
with open(file_path, 'r', encoding='utf-8') as infile:
|
|
for line_number, line in enumerate(infile, start=1):
|
|
line = line.strip()
|
|
if not line:
|
|
continue
|
|
try:
|
|
json_obj = json.loads(line)
|
|
context = json_obj.get('context')
|
|
if context and context not in unique_contexts_dict:
|
|
unique_contexts_dict[context] = None
|
|
except json.JSONDecodeError as e:
|
|
print(f"JSON decoding error in file {filename} at line {line_number}: {e}")
|
|
except FileNotFoundError:
|
|
print(f"File not found: {filename}")
|
|
continue
|
|
except Exception as e:
|
|
print(f"An error occurred while processing file {filename}: {e}")
|
|
continue
|
|
|
|
unique_contexts_list = list(unique_contexts_dict.keys())
|
|
print(f"There are {len(unique_contexts_list)} unique `context` entries in the file {filename}.")
|
|
|
|
try:
|
|
with open(output_path, 'w', encoding='utf-8') as outfile:
|
|
json.dump(unique_contexts_list, outfile, ensure_ascii=False, indent=4)
|
|
print(f"Unique `context` entries have been saved to: {output_filename}")
|
|
except Exception as e:
|
|
print(f"An error occurred while saving to the file {output_filename}: {e}")
|
|
|
|
print("All files have been processed.")
|
|
```
|
|
|
|
### Step-1 Insert Contexts
|
|
|
|
Insert the extracted contexts into the LightRAG system.
|
|
|
|
**Code**
|
|
|
|
```python
|
|
def insert_text(rag, file_path):
|
|
with open(file_path, mode='r') as f:
|
|
unique_contexts = json.load(f)
|
|
|
|
retries = 0
|
|
max_retries = 3
|
|
while retries < max_retries:
|
|
try:
|
|
rag.insert(unique_contexts)
|
|
break
|
|
except Exception as e:
|
|
retries += 1
|
|
print(f"Insertion failed, retrying ({retries}/{max_retries}), error: {e}")
|
|
time.sleep(10)
|
|
if retries == max_retries:
|
|
print("Insertion failed after exceeding the maximum number of retries")
|
|
```
|
|
|
|
### Step-2 Generate Queries
|
|
|
|
Extract tokens from the first and second half of each context, then combine them as dataset descriptions to generate queries.
|
|
|
|
**Code**
|
|
|
|
```python
|
|
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
|
|
|
|
def get_summary(context, tot_tokens=2000):
|
|
tokens = tokenizer.tokenize(context)
|
|
half_tokens = tot_tokens // 2
|
|
|
|
start_tokens = tokens[1000:1000 + half_tokens]
|
|
end_tokens = tokens[-(1000 + half_tokens):1000]
|
|
|
|
summary_tokens = start_tokens + end_tokens
|
|
summary = tokenizer.convert_tokens_to_string(summary_tokens)
|
|
|
|
return summary
|
|
```
|
|
|
|
### Step-3 Query
|
|
|
|
Extract and query LightRAG with the queries generated in Step-2.
|
|
|
|
**Code**
|
|
|
|
```python
|
|
def extract_queries(file_path):
|
|
with open(file_path, 'r') as f:
|
|
data = f.read()
|
|
|
|
data = data.replace('**', '')
|
|
|
|
queries = re.findall(r'- Question \d+: (.+)', data)
|
|
|
|
return queries
|
|
```
|